"bzr: ERROR: exceptions.AssertionError: second push failed to complete a fetch set" during checkout

[Francis Devereux]

There is quite a lot of background information on this error, so please bear with me.

My company use Subversion for version control. For the past month or so I have been using bzr and bzr-svn to interact with the svn repository and I have several Bazaar branches on my machine. I now need to let another developer work on one of my branches, so I have been setting up a Bazaar repository on a server and pushing my branch there.

On my laptop (OS X 10.5.8, Bazaar 2.0.2 and bzr-svn 1.0.1 from Macports + various other plugins) I have the following branches inside a shared repository:
asset-bank-trunk: A bzr checkout of the trunk of the svn repository
asset-bank-attval-perfenh: a branch of asset-bank-trunk containing the feature I want to share with another developer
asset-bank-cleanups: a branch of asset-bank-trunk containing small changes I want to push back to the svn trunk often
asset-bank-release: another branch of asset-bank-trunk

I installed bazaar 2.0.2 on one of our servers (ringmer, Debian Lenny) by compiling and installing Debian source packages from testing (I wanted to use a newer version than the one available in Lenny). I created a repository using something like the following (these commands are from memory, not copy/pasted):
cd /home/bzr/repos
bzr init-repo asset-bank --2a --no-trees
cd asset-bank
bzr co svn+ssh://<our-svn-trunk> trunk
mkdir branches
bzr branch trunk branches/attval-perfenh

I decided to rename the checkout of the svn trunk from trunk to svn-trunk so that if we migrate from svn to bzr we can create another bzr branch called "trunk" to use as the master branch:
mv trunk svn-trunk
# edited parent-location in branches/attval-perfenh/.bzr/branch/branch.conf

I then pushed the changes on my laptop to the server and then got a new checkout and made some commits:
cd asset-bank-attval-perfenh
bzr push bzr+ssh://ringmer.<our-domain>/home/bzr/repos/asset-bank/branches/attval-perfenh
cd ..
bzr co bzr+ssh://ringmer.bsvr.co.uk/home/bzr/repos/asset-bank/branches/attval-perfenh
cd attval-perfenh
# do some coding
bzr ci

Everything seemed to work fine up to this point. I then decided to do a checkout in a Windows VM to check that it worked because the developer I'll be working with uses Windows. I used TortoiseBzr installed using the Bazaar 2.0.2 installer for this:

1) Used TortoiseBzr to create a new shared repository
2) Tried to check out the attval-perfenh branch from the server. This failed with the following error:
Connected (version 2.0, client OpenSSH_5.1p1)
Authentication (publickey) successful!
Secsh channel 1 opened.
bzr: ERROR: exceptions.AssertionError: second push failed to complete a fetch set([('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>')]).

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 842, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1037, in run_bzr
  File "bzrlib\commands.pyo", line 654, in run_argv_aliases
  File "C:/Program Files/Development/Bazaar/plugins\qbzr\lib\commands.py", line 787, in run
  File "bzrlib\commands.pyo", line 1037, in run_bzr
  File "bzrlib\commands.pyo", line 654, in run_argv_aliases
  File "bzrlib\builtins.pyo", line 1336, in run
  File "bzrlib\branch.pyo", line 1330, in create_checkout
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\branch.pyo", line 948, in pull
  File "bzrlib\branch.pyo", line 3194, in pull
  File "bzrlib\branch.pyo", line 3071, in pull
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\branch.pyo", line 896, in update_revisions
  File "bzrlib\branch.pyo", line 3014, in update_revisions
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\branch.pyo", line 579, in fetch
  File "bzrlib\repository.pyo", line 1695, in fetch
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\repository.pyo", line 3413, in fetch
  File "bzrlib\fetch.pyo", line 81, in __init__
  File "bzrlib\fetch.pyo", line 107, in __fetch
  File "bzrlib\fetch.pyo", line 148, in _fetch_everything_for_search
AssertionError: second push failed to complete a fetch set([('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>')]).

bzr 2.0.2 on python 2.5.4 (Windows-XP-5.1.2600-SP3)
arguments: ['C:\\Program Files\\Development\\Bazaar\\bzr.exe', 'qsubprocess', '--bencode', 'l8:checkout79:bzr+ssh://ringmer.<our-domain>/home/bzr/repos/asset-bank/branches/attval-perfenh/39:C:\\src\\bright\\asset-bank\\attval-perfenhe']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files\Development\Bazaar\plugins\bzrtools [2.0.1]
  explorer C:\Program Files\Development\Bazaar\plugins\explorer [0.8.3]
  launchpad C:\Program Files\Development\Bazaar\plugins\launchpad [2.0.2]
  netrc_credential_store C:\Program Files\Development\Bazaar\plugins\netrc_credential_store [2.0.2]
  qbzr C:\Program Files\Development\Bazaar\plugins\qbzr [0.14.5]
  rebase C:\Program Files\Development\Bazaar\plugins\rebase [0.5.4]
  svn C:\Program Files\Development\Bazaar\plugins\svn [1.0.0]
  upload C:\Program Files\Development\Bazaar\plugins\upload [1.0.0dev]
  xmloutput C:\Program Files\Development\Bazaar\plugins\xmloutput [0.8.5]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.

I have had a look at some of the revids in the error message on my laptop and they seem to be in the other branches on my laptop. For example:
francis@spaceman attval-perfenh $ bzr log -rrevid:<email address hidden>
------------------------------------------------------------
revno: 3522.1.3
committer: Francis Devereux <email address hidden>
branch nick: asset-bank-release
timestamp: Wed 2009-10-21 12:18:25 +0100
message:
  Renamed project-switch-deploy.xml to build.xml

I thought that perhaps if I created some more branches in the shared repository on the server and pushed the branches from my laptop into them that might resolve the problem, but it didn't (although the revids in the error message changed)
# laptop (after creating branches on server)
cd asset-bank-release/
bzr push bzr+ssh://ringmer.<our-domain>/home/bzr/repos/asset-bank/branches/francis/release
cd ../asset-bank-cleanups/
bzr push bzr+ssh://ringmer.<our-domain>/home/bzr/repos/asset-bank/branches/francis/cleanups
cd ../asset-bank-trunk/
bzr push bzr+ssh://ringmer.<our-domain>/home/bzr/repos/asset-bank/branches/francis/trunk

After this I deleted the shared repository and failed checkout in the Windows XP VM and tried the checkout again. Here is the error message I got this time:
Connected (version 2.0, client OpenSSH_5.1p1)
Authentication (publickey) successful!
Secsh channel 1 opened.
bzr: ERROR: exceptions.AssertionError: second push failed to complete a fetch set([('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>')]).

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 842, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1037, in run_bzr
  File "bzrlib\commands.pyo", line 654, in run_argv_aliases
  File "C:/Program Files/Development/Bazaar/plugins\qbzr\lib\commands.py", line 787, in run
  File "bzrlib\commands.pyo", line 1037, in run_bzr
  File "bzrlib\commands.pyo", line 654, in run_argv_aliases
  File "bzrlib\builtins.pyo", line 1336, in run
  File "bzrlib\branch.pyo", line 1330, in create_checkout
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\branch.pyo", line 948, in pull
  File "bzrlib\branch.pyo", line 3194, in pull
  File "bzrlib\branch.pyo", line 3071, in pull
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\branch.pyo", line 896, in update_revisions
  File "bzrlib\branch.pyo", line 3014, in update_revisions
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\branch.pyo", line 579, in fetch
  File "bzrlib\repository.pyo", line 1695, in fetch
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\repository.pyo", line 3413, in fetch
  File "bzrlib\fetch.pyo", line 81, in __init__
  File "bzrlib\fetch.pyo", line 107, in __fetch
  File "bzrlib\fetch.pyo", line 148, in _fetch_everything_for_search
AssertionError: second push failed to complete a fetch set([('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>')]).

bzr 2.0.2 on python 2.5.4 (Windows-XP-5.1.2600-SP3)
arguments: ['C:\\Program Files\\Development\\Bazaar\\bzr.exe', 'qsubprocess', '--bencode', 'l8:checkout79:bzr+ssh://ringmer.<our-domain>/home/bzr/repos/asset-bank/branches/attval-perfenh/39:C:\\src\\bright\\asset-bank\\attval-perfenhe']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files\Development\Bazaar\plugins\bzrtools [2.0.1]
  explorer C:\Program Files\Development\Bazaar\plugins\explorer [0.8.3]
  launchpad C:\Program Files\Development\Bazaar\plugins\launchpad [2.0.2]
  netrc_credential_store C:\Program Files\Development\Bazaar\plugins\netrc_credential_store [2.0.2]
  qbzr C:\Program Files\Development\Bazaar\plugins\qbzr [0.14.5]
  rebase C:\Program Files\Development\Bazaar\plugins\rebase [0.5.4]
  svn C:\Program Files\Development\Bazaar\plugins\svn [1.0.0]
  upload C:\Program Files\Development\Bazaar\plugins\upload [1.0.0dev]
  xmloutput C:\Program Files\Development\Bazaar\plugins\xmloutput [0.8.5]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.

[Francis Devereux (frankoid)]

I have worked around the problem by manually copying the missing revisions to the server using the script below. I think that it is a bug that they weren't copied automatically when I did "bzr push bzr+ssh://ringmer.<our-domain>/home/bzr/repos/asset-bank/branches/attval-perfenh" though.

#!/opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python
from bzrlib.repository import Repository

REVIDS=['<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>']

remote_repo = Repository.open('bzr+ssh://ringmer.<our-domain>/home/bzr/repos/asset-bank')
local_repo = Repository.open('/Users/francis/Code/Bright/asset-bank')

for revid in REVIDS:
    remote_repo.fetch(local_repo, revision_id=revid)

print "done"

[Andrew Bennetts (spiv)]

This bears some resemblance to bug 437626, but it appears to be a different bug because a) it happens with 2a repositories, and b) bug 437626 is fixed in bzr 2.0.2, which the reporter is using.

I'm pretty concerned that this is happening. Also, experience from bug 437626 suggests that it will be quite hard to reproduce and thus diagnose. Is it possible to get a tarball of the repository/repositories that we can use to reproduce the error? I see you say you've manually repaired the repository by fetching some revisions manually, if you don't have an unrepaired copy around that still reproduces, then tarball taken as soon as possible after the repair may still help.

[Andrew Bennetts (spiv)]

The obvious thing to be suspicious of I suppose is the implementation of GroupCompressVersionedFiles.insert_record_stream, and its interaction with _GCGraphIndex.add_records (i.e. perhaps essentially the same bug as bug 437626, but in the groupcompress code rather than the knit code). At a glance I don't see any obvious flaw, but it's also not obviously correct to me either.

[Francis Devereux (frankoid)]

It probably won't be possible to give you a copy of our repository I'm afraid. However I have managed to reproduce the bug by performing the same steps again (create repo on server, check out svn, branch, push to server branch from laptop, attempt to check out server branch without using a shared repo) so I will be able to test any fixes that you come up with.

I will let you know if I manage to reproduce the bug with a test repository that I can share.

Not sure if this is relevant but I have been using "bzr rebase" with some of the branches on my laptop before pushing the changes to the svn trunk. In particular I have done this several times with the "asset-bank-cleanups" branch but I am pretty sure I have never done it with the "asset-bank-attval-perfenh" branch (because I haven't pushed from asset-bank-attval-perfenh to svn trunk). I have also cherry picked some revisions from asset-bank-attval-perfenh into svn trunk (general fixes that are not part of the feature being developed on that branch).

I also seemed to hit bug #437003 when doing a checkin to the attval-perfenh bound branch this morning, but was able to work around it with "bzr pack".