"Unrecognised container format" after push to FTP server with broken APPE

[enobrev]

On my server, I'm running bzr 1.7. My client on my windows system is 1.7. My co-developer's client on a mac system is the latest stable as of last week (I'll find out when he logs on again).

When I push to my server, everything works fine, well and great. This has held true for many projects over the last year or so, after going through upgrades on both the server and my client.

When my co-developer pushes to the server, the repository gets corrupted. I'd seen this once before, but merely killed the repo and repushed when I couldn't find a solution in the past. This time I'd like to resolve the issue if possible.

When I try a bzr log (or update or much of anything) I get the following message:

bzr: ERROR: Unrecognised container format: 'B1109'

Per the advice of the kind folk at #bzr it tried:
for i in .bzr/repository/packs/*.pack; do echo $i; head -n 2 $i; done

Which shows that 2 of the files do not have their proper header:

.bzr/repository/packs/0792545519082d4c470d215df4bae156.pack
Bazaar pack format 1 (introduced in 0.18)
B392
.bzr/repository/packs/0b839c4de16a81463ccd73587ba7a8d7.pack
B1109

.bzr/repository/packs/25eeb29c1d1f33452f187d0db4d6b15c.pack
Bazaar pack format 1 (introduced in 0.18)
B371

...

.bzr/repository/packs/83f4b2bef2a6c446a38853b6a81d5fa5.pack
B363

.bzr/repository/packs/982417c4bac61a0fefa7f83485c33088.pack
Bazaar pack format 1 (introduced in 0.18)
B367

I've also noticed something else that's a bit odd. If I list the files in the pack directory, the 2 files without their proper header have different permissions than the others:

-rwxr-xr-x 1 enobrev.info enobrev.info 4616 Dec 3 06:59 83f4b2bef2a6c446a38853b6a81d5fa5.pack
-rw-r--r-- 1 enobrev.info enobrev.info 15711 Nov 30 04:33 c68a2940591c95c9142716739d0d0b2d.pack
-rw-r--r-- 1 enobrev.info enobrev.info 2454468 Nov 30 02:46 76b68498954878e4addce109ccccc913.pack
-rwxr-xr-x 1 enobrev.info enobrev.info 259643 Dec 3 06:59 0b839c4de16a81463ccd73587ba7a8d7.pack

I tried adding the header myself via vim, but then I get:

bzr: ERROR: Unknown record type: '\xc7'

Here's the Trace Log:

2.102 Traceback (most recent call last):
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/commands.py", line 857, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/builtins.py", line 2399, in run
    check_dwim(path, verbose, do_branch=branch, do_repo=repo, do_tree=tree)
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/check.py", line 302, in check_dwim
    result = repo.check()
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/repository.py", line 1850, in check
    return self._check(revision_ids)
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/repository.py", line 1854, in _check
    result.check()
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/check.py", line 76, in check
    self.check_revision_graph()
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/check.py", line 97, in check_revision_graph
    bad_revisions = self.repository._find_inconsistent_revision_parents()
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/repofmt/pack_repo.py", line 1768, in _find_inconsistent_revision_parents
    revs = self.get_revisions(rev_ids)
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/repository.py", line 1123, in get_revisions
    return self._get_revisions(revision_ids)
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/repository.py", line 1134, in _get_revisions
    for record in stream:
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/knit.py", line 1221, in get_record_stream
    needed_from_fallback - absent_keys)
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/knit.py", line 992, in _get_content_maps
    record_map = self._get_record_map(keys, allow_missing=True)
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/knit.py", line 1116, in _get_record_map
    self._read_records_iter(records):
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/knit.py", line 1563, in _read_records_iter
    izip(iter(needed_records), raw_data):
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/knit.py", line 2408, in get_raw_records
    for names, read_func in reader.iter_records():
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/pack.py", line 253, in iter_records
    self._read_format()
  File "/home/16139/users/.home/usr/lib/python2.5/bzrlib/pack.py", line 296, in _read_format
    raise errors.UnknownContainerFormatError(format)
UnknownContainerFormatError: Unrecognised container format: 'B1109'

2.102 return code 3

I can't find any info about this anywhere. Please help!!

Mark

[John A Meinel (jameinel)]

There was a bug in an old version of paramiko such that doing an "append" actually caused a write to occur at the beginning of the file.

the "B1109" is saying that what follows is a byte-stream of length 1109. And it sounds like whatever he is doing is causing the pack header to either

a) Not be written, or
b) Get overwritten

What does he use to do the his push? (bzr+ssh, network mount, sftp, etc?)

I would guess he hasn't corrupted any existing packs, because we don't write into them. Though it is possible his client wanted to do an autopack and would copy data across, etc.

Certainly something serious seems to be happening if we are saying "Append these 50 bytes" and it is being turned into "write these 50 bytes at the beginning of the file".

[enobrev (enobrev)]

He did a pull, ci, and push.

He's using ftp. Also, I was mistaken, he's using a linux client.

I'm not sure what to do from here. I could replace the repo with mine and let him push again, but that's a band-aid. Please let me know how I can help fix this.

Mark

[enobrev (enobrev)]

Just realized he has an old version and asked that he upgrade. Hopefully that resolves the issue.

Client Version:

Bazaar (bzr) 1.3.1
Python interpreter: /usr/bin/python 2.4.3.final.0
Python standard library: /usr/lib/python2.4
bzrlib: /usr/lib/python2.4/site-packages/bzrlib

Server Version

Bazaar (bzr) 1.7
Python interpreter: /usr/bin/python 2.5.1
Python standard library: /usr/lib/python
bzrlib: /usr/lib/python/bzrlib

Mark

[enobrev (enobrev)]

Ok, the client and server are now 1.9.

I reset the branch entirely by removing the directory, recreating it and doing a bzr init and then asked the developer to do a push, so his would be the only one. The problem persists. I asked him to list the headers on his system and they all seem correct, so it's something happening during the push.

He's also upgraded paramiko, though I'm not sure that would make a difference since he's using ftp. Still having the problem.

Any suggestions? Any way I could help debug?

Thanks!

Mark

[Andrew Bennetts (spiv)]

> He's also upgraded paramiko, though I'm not sure that would make a
> difference since he's using ftp. Still having the problem.
>
> Any suggestions? Any way I could help debug?

Which FTP server is it? My guess is that it's a bug in the FTP server. bzr
would be sending “APPE” commands to append data, but it sounds like your FTP
server is overwriting instead of appending.

-Andrew.

[enobrev (enobrev)]

I'm not sure yet, but I have narrowed it down to ftp only. I tried pushing from my system via ftp rather than sftp and saw the same problem. The server in question is Media Temple Grid Server. Will see if I can find out what specific ftp server and version they're running.

[John A Meinel (jameinel)]

I'm guessing the problem is that your FTP server claims to support "APPE" (append) but then doesn't actually append the bytes to the file.

I really don't have anything to offer other than access via something other than FTP. There isn't really much we can do about a system that claims to support append and then doesn't do it.

[Jonas Schwabe (jonas-schwabe)]

My problem is simmlear to these...
bzr: ERROR: Unrecognised container format: 'B40294'
useing FTP, too!
The FTP Server is of a free Webspace (http://www.bplaced.net)... Will test one other!

[Dominic Hosler (dominichosler)]

I realise this is an old bug but it still exists for me.

If the cause is the APPE command not being implemented properly on the server side ftp.
This problem was fixed in version 1.18, see bug https://bugs.launchpad.net/bzr/+bug/294709.

I don't understand why the problem still exists, unless there's a regression, I'm using versions Bazaar Explorer 1.1b1, QBzr 0.19.0, bzrlib 2.2.0.

I get the same error but with a different number 'B508' related to the number of commits in my local repository.

[Martin Pool (mbp)]

Hi Dominic, gh1234,

bug 294709 says we should cope when the server says it doesn't support APPE.

From what's been posted to these bugs it seems like the server says it supports APPE but actually gets it wrong. (It's a bit hard to believe that's really true...)

I guess we could add a configuration option that tells bzr not to use APPE even though the server claims to support it, thereby invoking the logic in bug 294709. It will be a bit slow, but it would work.

[Martin (martin22)]

Hi,

I am very much blocked by a similar bug (.bzr.log attached). It perfectly worked a few weeks ago, I guess the FTP server (pure-ftpd) has somehow changed.

Is there a solution so that I can push again?

Regards, --Martin

Error:
$ bzr push
Using saved push location: ftp://ftp.server.net/www/me/soft/dev/
bzr: ERROR: Unrecognised container format: 'B1230'

[Andrew Bennetts (spiv)]

Hi Martin,

The solution is to use something other than FTP, or fix the FTP server to be one that deals with APPE commands correctly (or at least doesn't claim to support APPE).

There's not much bzr can do about servers that claim to support a feature but that then silently write corrupt data when you use that feature. The best we could do is add a configuration option to bzr so that you can force it to never use APPE (and be much slower), as Martin Pool says in comment 10.

If someone is interested in writing the code to add that configuration option we'd be happy to mentor the effort.

[Martin (martin22)]

Hi Andrew,

Many thanks for the answer.

> The solution is to use something other than FTP, or fix the
> FTP server to be one that deals with APPE commands
> correctly (or at least doesn't claim to support APPE).
In this particular case, I have no way to use something else than FTP (Bazaar can't work in CGI mode, right?) and I am not admin of the FTP server (and no easy way to ask :-(

According to the log file, are you sure it's also related to a bug in APPE on the server side?

Best regards, --Martin