Pushing to shared repo using FTP doesn't work: "Append/Restart not permitted, try again"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| Bazaar |
Medium
|
Unassigned | ||
| Tahoe-LAFS |
Invalid
|
Undecided
|
Unassigned |
Bug Description
I just created a shared repository using an FTP location:
bzr init-repo --rich-root-pack --no-trees ftp://<email address hidden>/bzr-repo
That worked fine.
Afterwards, I tried to push a branch 'proj1' to this newly created repo:
bzr push --remember ftp://<email address hidden>
That command failed with a:
FTP temporary error: 451 /bzr-repo/
[...]
bzr: ERROR: Transport error: FTP temporary error during APPEND /bzr-repo/
See attached .bzr.log for more details.
I am using Bazaar 1.8.
Best regards,
Jonny
Related branches
- Martin Pool: Needs Fixing on 2009-07-20
- Diff: 68 lines
Jonny Dee (jonny.dee) wrote : | #3 |
Dan Watkins (oddbloke) wrote : Re: [Bug 294709] [NEW] Pushing to shared repo using FTP doesn't work: "Append/Restart not permitted, try again" | #4 |
Robert Collins (lifeless) wrote : Re: [Bug 294709] [NEW] Pushing to shared repo using FTP doesn't work: "Append/Restart not permitted, try again" | #5 |
On Thu, 2008-11-06 at 14:49 +0000, Daniel Watkins wrote:
>
> > bzr: ERROR: Transport error: FTP temporary error during
> APPEND /bzr-repo/
> It seems like we should be able to work around this by doing 'get
> remote
> file', 'perform append locally, 'push new file'.
We don't actually append to any files. The .fetch file is streamed out
by the push command - it starts empty.
This is the FTP implementation of a stream expecting APPE to work (which
is a reasonable expectation :() and being denied.
So - its a FTP transport bug, and probably what the FTP transport needs
to do is to write the first block of data its given in two parts
(keeping both in memory) so it can be sure that APPE works; if it
doesn't, make a temp file and buffer, if it does, cool.
Note that using a temp file will require redirecting read/readv requests
to the streams name into the local temp file.
status confirmed
importance medium
-Rob
--
GPG key available at: <http://
Changed in bzr: | |
importance: | Undecided → Medium |
status: | New → Confirmed |
Peter Koppatz (peter-koppatz) wrote : | #6 |
I had a similar problem and solved it by using proftpd and adding/setting the parameter "AllowStoreRestart on".
Aldo "xoen" Giambelluca (xoen) wrote : | #7 |
Hi everybudy, first : THANK YOU for your work ;).
I tried to push a repository online and I have the same problem.
I can't change things like FTP server and so on, I can only use FTP, I need this to work with my friend on something, is there a way? a workaround?
Martin Pool (mbp) wrote : Re: [Bug 294709] Re: Pushing to shared repo using FTP doesn't work: "Append/Restart not permitted, try again" | #8 |
2009/4/29 Aldo "xoen" Giambelluca <email address hidden>:
> Hi everybudy, first : THANK YOU for your work ;).
>
> I tried to push a repository online and I have the same problem.
> I can't change things like FTP server and so on, I can only use FTP, I need this to work with my friend on something, is there a way? a workaround?
Hi Aldo,
Unfortunately if you can't change the server I don't think there is a
workaround until this is fixed in Bazaar. If you're interested in
making a patch as described by Robert it should not be very hard and
we will review it for you.
--
Martin <http://
Nils Durner (ndurner) wrote : | #9 |
Hi,
the attached patch adds a fallback to GET + PUT if APPE fails.
Tested against Twisted 8.2.0 FTP Server.
Best,
Nils Durner
Martin Pool (mbp) wrote : | #10 |
2009/7/14 Nils Durner <email address hidden>:
> Hi,
>
> the attached patch adds a fallback to GET + PUT if APPE fails.
> Tested against Twisted 8.2.0 FTP Server.
Hi Nils,
Thanks for the patch. Could you please put it in a branch on
Launchpad and create a merge proposal so that we can review and track
it?
The patch looks reasonable to me. In some cases this may be a lot
slower so we should at least write a warning to the user the first
time this happens.
It would be nice to add a test that it works correctly, using our test
suite's medusa ftp server. I wouldn't strictly insist on having one
but at least adding a TODO to the test file would be good.
--
Martin <http://
Zooko Wilcox-O'Hearn (zooko) wrote : | #11 |
Nils: How does this issue affect TahoeLAFS?
Nils Durner (ndurner) wrote : | #12 |
@Martin: ok!
@Zooko: Bazaar uses extended FTP commands like APPE and SITE which are not supported by some FTP servers (like the one that is integrated in Tahoe). With my patch applied, users can store their Bazaar repositories in a Tahoe storage grid (aka "SCM in the cloud").
Zooko Wilcox-O'Hearn (zooko) wrote : | #13 |
Nils: Awesome! Thank you very much for this patch. If it gets accepted into bzr in time for the TahoeLAFS v1.5 release (due any day now), then I'll include the fact of its existence in the TahoeLAFS release announcement! :-)
John A Meinel (jameinel) wrote : | #14 |
I'll just mention that higher level code operations assume things about being able to append. Which means that while your code will probably work, I believe it will become quadratic if you try to push up a large branch.
1) Push up rev 1
2) Download rev 1 + push up rev 1 & 2
3) Download rev 1&2, push up rev 1-3
4) Download rev 1-3, push up rev 1-4
...
100) Download revs 1-99, push up revs 1-100
Now we do a little bit with buffering, so we should only append XX data per attempt, but I'm curious if supporting something with abysmal performance is worthwhile...
Nils Durner (ndurner) wrote : | #15 |
I agree that large pushes are problematic. But depending on your workflow, small pushes may be much more common than really big (and thus expensive) ones. This is especially true for solo repositories that are usually always reachable: one push (pack) would virtually always consist of just one commit.
Martin Pool (mbp) wrote : | #16 |
Patch sent to pqm
Changed in bzr: | |
status: | Confirmed → In Progress |
Martin Pool (mbp) wrote : | #17 |
See bug 409615 - we can probably eliminate at least some cases where we're unnecessarily asking to append when creating a new file, which may reduce/eliminate the need to use this fallback. Want to try, Nils?
Changed in bzr: | |
status: | In Progress → Fix Released |
milestone: | none → 1.18 |
Peter Liepa (pliepa) wrote : | #18 |
In duplicate Bug #409507, Martin Pool asked: "Do you know what (FTP) server software it is?"
I checked with my hosting provider and they said
"Our servers use proftpd. http://
I'm really impressed with how fast you've replied to my bug submission. I guess I'll keep an
eye out for v1.18.
I'm a little surprised that you haven't hit this sever problem long ago, but then again I
know next to nothing about how FTP works and how many server implementations there are.
John A Meinel (jameinel) wrote : Re: [Bug 294709] Re: Pushing to shared repo using FTP doesn't work: "Append/Restart not permitted, try again" | #19 |
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Peter Liepa wrote:
> In duplicate Bug #409507, Martin Pool asked: "Do you know what (FTP)
> server software it is?"
>
> I checked with my hosting provider and they said
>
> "Our servers use proftpd. http://
>
> I'm really impressed with how fast you've replied to my bug submission. I guess I'll keep an
> eye out for v1.18.
>
> I'm a little surprised that you haven't hit this sever problem long ago, but then again I
> know next to nothing about how FTP works and how many server implementations there are.
>
We've certainly hit servers that didn't support APPE (append) before.
Just that nobody cared enough to implement support for it. As usually
switching to sftp is a *much* better decision for a large variety of
reasons. (FTP also doesn't support partial *reads* which causes some
significant performance implications when fetching data from a repository.)
FTP as a protocol is not a rich protocol, and not particularly secure
(passwords are transmitted in plain-text across the wire).
It is supported, because it was possible to do so, but it certainly
isn't something we would specifically *recommend*.
John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://
iEYEARECAAYFAkp
alMAoNmMeQDanHW
=0Du8
-----END PGP SIGNATURE-----
Vincent Ladeuil (vila) wrote : | #20 |
>>>>> Peter Liepa writes:
> I checked with my hosting provider and they said
> "Our servers use proftpd. http://
proftpd certainly support APPE. It may be configured to *not*
support it though and that may be the case for the server you're
using, you may want to ask "them" why "they" did so though...
Peter Liepa (pliepa) wrote : | #21 |
John A Meinel wrote:
>We've certainly hit servers that didn't support APPE (append) before.
>Just that nobody cared enough to implement support for it. As usually
>switching to sftp is a *much* better decision for a large variety of
>reasons [...]
Ok, thanks, the docs could be clearer about this, and maybe bazaar could
avoid crashing. I'm checking with my hosting
provider whether SFTP is available (probably is, I've just never used it before,
and am having trouble getting a successful "hello world" connection). But
that sounds like the way to go. From an above comment, maybe the
proftpd server has AllowStoreRestart set the wrong way, but I doubt I
could get them to change it.
Martin Pool (mbp) wrote : Re: [Bug 294709] Re: Pushing to shared repo using FTP doesn't work: "Append/Restart not permitted, try again" | #22 |
2009/8/8 Peter Liepa <email address hidden>:
> John A Meinel wrote:
>>We've certainly hit servers that didn't support APPE (append) before.
>>Just that nobody cared enough to implement support for it. As usually
>>switching to sftp is a *much* better decision for a large variety of
>>reasons [...]
>
> Ok, thanks, the docs could be clearer about this, and maybe bazaar could
> avoid crashing.
Right, from the next release of bzr we will give a message and continue.
Where do you think this should be described in the Bazaar documentation?
--
Martin <http://
Peter Liepa (pliepa) wrote : | #23 |
There is a user guide at http://
My hosting service has disabled proftpd APPE, rightly or wrongly, because of the possibility of malicious use. Also, my cheapo plan does not support SSH or SFTP, and I'm unwilling to pay the extra $ to upgrade. Also, our volunteer-run website will soon be migrating to a server that does not support SSH at all, or so I'm told. I can understand that the bazaar project may be concentrating on serving a more savvy and professional grade of user than myself. I'm going to try springloops.com, because they have a different model of FTP deployment - but I'd be more than happy to revisit bazaar when the limitations of FTP have been eased.
In closing, I'd like to mention that FTP website deployment via VCS seems to be almost a "killer app" that is under-addressed for the masses like me. I'm a relative web newbie who doesn't have the resources or inclination to use SSH, configure Apache, install server-side VCS, write Python commit hooks, etc, etc, yet would love to deploy websites via FTP from a VCS in a platform independent (mac and pc) way. So far I haven't found a solution that addresses my needs, although I'll be looking at Bespin and Springloops. I'll also be keeping an eye on how you guys progress on this. Meanwhile, thanks for your rapid responses and helpful attention.
Martin Pool (mbp) wrote : | #24 |
FTP service is definitely a feature we want to support. It's true
that there seem to be more quirks in FTP servers than in SFTP and the
protocol is not quite as capable - understandable considering their
relative ages.
If you would like to try 1.18rc1 when it comes out and report on the
other bug (or a new bug) whether it works ok that would be
appreciated.
--
Martin <http://
Vincent Ladeuil (vila) wrote : Re: [Bug 294709] Re: Pushing to shared repo using FTP doesn't work: "Append/Restart not permitted, try again" | #25 |
>>>>> Peter Liepa <email address hidden> writes:
<snip/>
> My hosting service has disabled proftpd APPE, rightly or
> wrongly, because of the possibility of malicious use.
Shudder.
Have you tried contacting them to see if that could be activated
on a per-user or per-directory basis ?
> Also, my cheapo plan does not support SSH or SFTP, and I'm
> unwilling to pay the extra $ to upgrade.
Hmm, I think that answer my question above :-/
> I can understand that the bazaar project may be
> concentrating on serving a more savvy and professional
> grade of user than myself.
That's not how I feel about ftp at all. There are definitely
performance and security issues with ftp and *these* are the main
reasons why the ftp transport got less love.
But the truth is also that we have few bug reports about ftp
*users* and gives the (maybe wrong) impression that the existing
ftp servers are ok.
If propftpd doesn't allow APPE by default (but how many admins
knows about to override that ?), now, that kind of change the
game.
Indeed, with my local_test_server plugin I had to configure
proftpd like:
<Anonymous /tmp>
User %(user)s
UserAlias anonymous %(user)s
AllowOverwrite on
AllowStoreRe
<Limit ALL>
AllowAll
</Limit>
</Anonymous>
> In closing, I'd like to mention that FTP website deployment
> via VCS seems to be almost a "killer app" that is
> under-addressed for the masses like me.
bzr is committed to addres what we call the 'dumb servers' (as
opposed to the smart servers which use bzr itself) which only
provides a remote file system.
<snip/>
> So far I haven't found a solution that addresses my needs,
> although I'll be looking at Bespin and Springloops. I'll
> also be keeping an eye on how you guys progress on this.
> Meanwhile, thanks for your rapid responses and helpful
> attention.
Launchpad itself offers code hosting, did you check it ?
Peter Liepa (pliepa) wrote : | #26 |
I'm beginning to think that satisfying users like me may be counterproductive for your project.
To recap, I'm helping a small group of volunteers to maintain a ski club website. I'd like to set up version control on server A, and deploy the website content to server B. I don't mind whether A and B are the same or different. My main restriction is that my only access to server B is via FTP. And of course I want the VCS to do the deployment -- it should only update the files that have changed or been added, while deleting those files that have been deleted.
In researching this, I found descriptions of people setting up a VCS on server B, and using the working tree as the actual website content. I.e A and B are the same. That meant that deployment was pretty well automatic .. whenever you committed/pushed a change to the VCS, the working tree was updated, and so the website was automatically updated. But these people were able to use SSH and set up a VCS on their server. So I couldn't do what they were doing.
I was hoping that I could pull off this trick with Bazaar, because it handled FTP updates of the VCS. Yesterday, I tried making this happen on a host server that didn't have the FTP APPE disabled. This time, the push to server B was successful. Unfortunately, the working tree was not populated. There was a long message explaining this and why it was so, but nevertheless it was a showstopper for me.
Even if this worked, there would potentially be the problem that the .bzr directory is actually part of my website. Probably not a showstopper -- I'd have to configure the server to make that directory's contents inaccessible to HTTP.
Meanwhile, springloops.com works for me. The configuration in this case is that my web host is server B, and springloops is server A. I.e. A and B are different.
Springloops seems to be optimized for website development and deployment. Bazaar seems to be optimized for code base maintenance and release deployment. Although the two are philosophically similar, it may be that you have to optimize for one or the other.
So. getting back to my opening point, it may be counterproductive for you to try to address my problems. A few developers have commented that FTP is somewhat unworthy of their love, and I construe that to mean that making it "just work" may be a source of unending aggravation. And I don't quite understand what it is that does not allow you to populate a remote working directory, but without that I still wouldn't be able to use Bazaar for my purposes.
I guess you could turn my comments around and, rather than concentrate on building a VCS for code development and deployment, you could look at supporting workflows for website development and deployment as well, and that would be great. Mercurial has made a step in that direction - there is a plugin called FTPExtension, but rightly or wrongly I got the feeling it was too bleeding edge for my purposes.
Hope this helps.
John A Meinel (jameinel) wrote : | #27 |
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
...
> Meanwhile, springloops.com works for me. The configuration in this case
> is that my web host is server B, and springloops is server A. I.e. A
> and B are different.
>
> Springloops seems to be optimized for website development and
> deployment. Bazaar seems to be optimized for code base maintenance and
> release deployment. Although the two are philosophically similar, it
> may be that you have to optimize for one or the other.
You might want to look at "bzr-upload" which is a plugin specifically
for uploading websites to other locations via the bzr transport
mechanism. I think it fits pretty much exactly what you are looking for.
John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://
iEYEARECAAYFAkq
xCsAn3pSayUff8e
=xCin
-----END PGP SIGNATURE-----
Daira Hopwood (daira) wrote : | #28 |
'Invalid' only because it didn't require any change to Tahoe-LAFS.
Changed in tahoe-lafs: | |
status: | New → Invalid |
Nicola Lunghi (nicola.lunghi) wrote : | #29 |
This bug also affects me
bzr --version
Bazaar (bzr) 2.3b1
Python interpreter: /usr/bin/python 2.6.5
Python standard library: /usr/lib/python2.6
Platform: Linux-2.
bzrlib: /usr/lib/
The error that return the ftp server is 451
and the bzrlib ftp layer cannot detect it
the file attached is my .bzr-log (i removed ftp address and password and my home address for security....)
the relevant lines are
9.816 FTP stat: /htdocs/
10.045 FTP get: /htdocs/
10.333 FTP rm: /htdocs/
10.393 FTP get: /htdocs/
10.693 Using fetch logic to copy between CHKInventoryRep
10.695 FTP put: /htdocs/
11.164 FTP has check: z1lkk39vtt2ohon
11.221 FTP has: /htdocs/
11.278 FTP appe to /htdocs/
11.279 FTP appe (try 0) to /htdocs/
[ 4808] 2010-10-12 06:33:27.985 WARNING: FTP temporary error: 451 /htdocs/
11.452 Constructing FTP instance against ('ftp.ciccio.com', None, '++++++++', '********', False)
11.863 FTP appe (try 1) to /htdocs/
[ 4808] 2010-10-12 06:33:28.569 WARNING: FTP temporary error: 451 /htdocs/
12.036 Constructing FTP instance against ('ftp.ciccio.com', None, '++++++++', '********', False)
12.449 FTP appe (try 2) to /htdocs/
[ 4808] 2010-10-12 06:33:29.156 WARNING: FTP temporary error: 451 /htdocs/
12.623 Constructing FTP instance against ('ftp.ciccio.com', None, '++++++++', '********', False)
13.048 FTP appe (try 3) to /htdocs/
13.339 Traceback (most recent call last):
File "/usr/lib/
return the_callable(*args, **kwargs)
File "/usr/lib/
ret = run(*run_argv)
File "/usr/lib/
return self.run(
File "/usr/lib/
return self._operation
File "/usr/lib/
self.cleanups, self.func, *args, **kwargs)
File "/usr/lib/
result = func(*args, **kwargs)
...
Nicola Lunghi (nicola.lunghi) wrote : | #30 |
On Thu, 2008-11-06 at 14:35 +0000, Jonny Dee wrote: .bzr/repository /upload/ k2a94q9mf00jznz 50sct.fetch: Append/Restart not permitted, try again. Retrying. .bzr/repository /upload/ k2a94q9mf00jznz 50sct.fetch. Aborting. 451 /bzr-repo/ .bzr/repository /upload/ k2a94q9mf00jznz 50sct.fetch: Append/Restart not permitted, try again
> FTP temporary error: 451 /bzr-repo/
> [...]
> bzr: ERROR: Transport error: FTP temporary error during APPEND /bzr-repo/
It seems like we should be able to work around this by doing 'get remote
file', 'perform append locally, 'push new file'.