Failure to autopack because of 'missing inventories'

[John A Meinel]

After Andrews' recent fix
4661 Canonical.com Patch Queue Manager 2009-09-09 [merge]
     (andrew) Fail commit_write_group for 2a if any relevant inventory,
       chk_bytes or text reference is absent (#406687).

for bug #406687, I ended up with a repository, where the pack files had 'skewed'. Namely, it seemed that I had some inventories in one pack file, and the associated revisions in a different pack file. When autopack ran, it saw that it wanted to combine the pack file that had the revisions, but *not* the one that had the inventories.

Because autopack works across just the subset of pack files that look "too loose", the check ran and then raised a failure because it thought that the inventories were not present.

Doing a full 'bzr pack' on the affected repository worked fine, and meant that future pushes worked as well.

So workaround 1 is to manually pack the repo.
A possible fix is to change the 'do I have enough inventories' logic, so that it checks the indexes in the entire repository, and not just the pack files being generated.

Arguably, we may want to keep the inventories in the same pack files (meaning that even though we have them in pack A, we'll copy them to pack B to keep them 'together').

However, we don't have the texts for a given revision all in the same pack file, nor the chk pages, since those don't change every commit. So logically we have to handle the split anyway.

I'm thinking this is probably Critical, because if someone 'in the wild' encounters this, they really won't know what is wrong. We could downgrade it to 'High' because 'bzr pack' works around it, though.

Filing against bzr 2.0.x because Andrew's patch landed there.

[Andrew Bennetts (spiv)]

John A Meinel wrote:
[...]
> A possible fix is to change the 'do I have enough inventories' logic, so that
> it checks the indexes in the entire repository, and not just the pack files
> being generated.

Hmm, that's what it should be doing already:

        no_fallback_inv_index = self.repo.inventories._index

etc.

And looking at your original error report, it says:

“ValueError: We are missing inventories for revisions:”

Which is not an error from my new check (it would raise BzrCheckError, and
never uses that exact message). So I don't think my change is the culprit.

Instead it appears _copy_inventory_texts in GCCHKPacker is what's failing,
and at a glance it appears that indeed it assumes that the inventory for a
revision will be in the same set of packs being packed.

[John A Meinel (jameinel)]

I think the chance for this to occur is rare enough that we can downgrade the severity.
The error is a bit confusing, but you can 'bzr pack' to recover.

[Francis Devereux (frankoid)]

I encountered this bug in the repository that was previously affected by bug #489211. I was able to work around it by doing a "bzr pack" in the repository on the server though.

francis@spaceman attval-perfenh $ bzr ci WEB-INF/src/com/bright/assetbank/attribute/service/AttributeStorageManager.java
Committing to: bzr+ssh://ringmer.<our-domain>/home/bzr/repos/asset-bank/branches/attval-perfenh/
modified WEB-INF/src/com/bright/assetbank/attribute/service/AttributeStorageManager.java
Waiting for Emacs...
bzr: ERROR: bzrlib.errors.ErrorFromSmartServer: Error received from smart server: ('error', "We are missing inventories for revisions: [('<email address hidden>',)]")

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", line 842, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", line 1037, in run_bzr
    ret = run(*run_argv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", line 654, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/builtins.py", line 3058, in run
    exclude=safe_relpath_files(tree, exclude))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/workingtree_4.py", line 197, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/mutabletree.py", line 229, in commit
    *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commit.py", line 393, in commit
    self.branch.repository, new_revno, self.rev_id)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/branch.py", line 909, in import_last_revision_info
    self.repository.fetch(source_repo, revision_id=revid)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/remote.py", line 1270, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/repository.py", line 3413, in fetch
    pb=pb, find_ghosts=find_ghosts)
  File "/opt/local/...

[John A Meinel (jameinel)]

Assigning this to canonical-bazaar because it is triggering a fair number of failures on the package importer. (Something like 10-20 failures in the last month.)

[John A Meinel (jameinel)]

All of the failing packages were repacked and requeued. So they are back on the working list.

We could still do code-level improvements in bzr to change autopack to not fail under this circumstance. Though I'm guessing we should reduce the priority of working on this?

i don't think it still needs to be next-in-queue

[Robert Collins (lifeless)]

Ubuntu one just ran into these symptoms, it confused several folk and chewed up a bunch of engineer time until this existing bug was found. They are now trying a pack and we'll see if it fixes it.

[Martin Pool (mbp)]

This also bit mysql in sf 00015414 so I'm bumping it up to critical.

[GuilhemBichot (guilhem-bichot)]

I confirm that this fixes our case.

[John A Meinel (jameinel)]

Landed across the 2.x series (2.0.7, 2.1.?, 2.2.?, 2.3.1, 2.4b1)

[Vincent Ladeuil (vila)]

That's 2.1.4, 2.2.5 as presented in the milestone menu ;) Better keep track of them *now*

[Chris Coulson (chrisccoulson)]

I've just reported what looks like a similar issue to this (bug 752919, although the retracer marked it as a duplicate of what looks like a different bug). It just started happening tonight on a branch that I've been happily pushing to for months already

[John A Meinel (jameinel)]

Marking this as a bug against Launchpad, because they need to upgrade to either bzr-2.3.1 or bzr-2.2.5 to get this fix. Until then, people pushing to Launchpad will run into this, because the autopack is being done server-side. (eg bug #752919)

[Robert Collins (lifeless)]

We are running 2.5mumble now.