Failure to autopack because of 'missing inventories'

Bug #437003 reported by John A Meinel
86
This bug affects 14 people
Affects Status Importance Assigned to Milestone
Bazaar
Fix Released
Critical
John A Meinel
2.0
Fix Released
High
John A Meinel
Launchpad itself
Fix Released
High
Unassigned

Bug Description

After Andrews' recent fix
4661 Canonical.com Patch Queue Manager 2009-09-09 [merge]
     (andrew) Fail commit_write_group for 2a if any relevant inventory,
       chk_bytes or text reference is absent (#406687).

for bug #406687, I ended up with a repository, where the pack files had 'skewed'. Namely, it seemed that I had some inventories in one pack file, and the associated revisions in a different pack file. When autopack ran, it saw that it wanted to combine the pack file that had the revisions, but *not* the one that had the inventories.

Because autopack works across just the subset of pack files that look "too loose", the check ran and then raised a failure because it thought that the inventories were not present.

Doing a full 'bzr pack' on the affected repository worked fine, and meant that future pushes worked as well.

So workaround 1 is to manually pack the repo.
A possible fix is to change the 'do I have enough inventories' logic, so that it checks the indexes in the entire repository, and not just the pack files being generated.

Arguably, we may want to keep the inventories in the same pack files (meaning that even though we have them in pack A, we'll copy them to pack B to keep them 'together').

However, we don't have the texts for a given revision all in the same pack file, nor the chk pages, since those don't change every commit. So logically we have to handle the split anyway.

I'm thinking this is probably Critical, because if someone 'in the wild' encounters this, they really won't know what is wrong. We could downgrade it to 'High' because 'bzr pack' works around it, though.

Filing against bzr 2.0.x because Andrew's patch landed there.

Related branches

Revision history for this message
Andrew Bennetts (spiv) wrote : Re: [Bug 437003] [NEW] Failure to autopack because of 'missing inventories'

John A Meinel wrote:
[...]
> A possible fix is to change the 'do I have enough inventories' logic, so that
> it checks the indexes in the entire repository, and not just the pack files
> being generated.

Hmm, that's what it should be doing already:

        no_fallback_inv_index = self.repo.inventories._index

etc.

And looking at your original error report, it says:

“ValueError: We are missing inventories for revisions:”

Which is not an error from my new check (it would raise BzrCheckError, and
never uses that exact message). So I don't think my change is the culprit.

Instead it appears _copy_inventory_texts in GCCHKPacker is what's failing,
and at a glance it appears that indeed it assumes that the inventory for a
revision will be in the same set of packs being packed.

John A Meinel (jameinel)
Changed in bzr:
milestone: 2.0.1 → none
Revision history for this message
John A Meinel (jameinel) wrote :

I think the chance for this to occur is rare enough that we can downgrade the severity.
The error is a bit confusing, but you can 'bzr pack' to recover.

Changed in bzr:
importance: Critical → High
Revision history for this message
Francis Devereux (frankoid) wrote :
Download full text (6.4 KiB)

I encountered this bug in the repository that was previously affected by bug #489211. I was able to work around it by doing a "bzr pack" in the repository on the server though.

francis@spaceman attval-perfenh $ bzr ci WEB-INF/src/com/bright/assetbank/attribute/service/AttributeStorageManager.java
Committing to: bzr+ssh://ringmer.<our-domain>/home/bzr/repos/asset-bank/branches/attval-perfenh/
modified WEB-INF/src/com/bright/assetbank/attribute/service/AttributeStorageManager.java
Waiting for Emacs...
bzr: ERROR: bzrlib.errors.ErrorFromSmartServer: Error received from smart server: ('error', "We are missing inventories for revisions: [('<email address hidden>',)]")

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", line 842, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", line 1037, in run_bzr
    ret = run(*run_argv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", line 654, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/builtins.py", line 3058, in run
    exclude=safe_relpath_files(tree, exclude))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/workingtree_4.py", line 197, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/mutabletree.py", line 229, in commit
    *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commit.py", line 393, in commit
    self.branch.repository, new_revno, self.rev_id)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/branch.py", line 909, in import_last_revision_info
    self.repository.fetch(source_repo, revision_id=revid)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/remote.py", line 1270, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/repository.py", line 3413, in fetch
    pb=pb, find_ghosts=find_ghosts)
  File "/opt/local/...

Read more...

Revision history for this message
John A Meinel (jameinel) wrote :

Assigning this to canonical-bazaar because it is triggering a fair number of failures on the package importer. (Something like 10-20 failures in the last month.)

Changed in bzr:
assignee: nobody → canonical-bazaar (canonical-bazaar)
assignee: canonical-bazaar (canonical-bazaar) → nobody
John A Meinel (jameinel)
Changed in bzr:
assignee: nobody → canonical-bazaar (canonical-bazaar)
Revision history for this message
John A Meinel (jameinel) wrote :

All of the failing packages were repacked and requeued. So they are back on the working list.

We could still do code-level improvements in bzr to change autopack to not fail under this circumstance. Though I'm guessing we should reduce the priority of working on this?

i don't think it still needs to be next-in-queue

Jelmer Vernooij (jelmer)
tags: added: missing-inventory packs
Revision history for this message
Robert Collins (lifeless) wrote :

Ubuntu one just ran into these symptoms, it confused several folk and chewed up a bunch of engineer time until this existing bug was found. They are now trying a pack and we'll see if it fixes it.

Martin Pool (mbp)
tags: added: mysql
Revision history for this message
Martin Pool (mbp) wrote :

This also bit mysql in sf 00015414 so I'm bumping it up to critical.

Changed in bzr:
importance: High → Critical
John A Meinel (jameinel)
Changed in bzr:
assignee: canonical-bazaar (canonical-bazaar) → John A Meinel (jameinel)
John A Meinel (jameinel)
Changed in bzr:
status: Confirmed → In Progress
Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

I confirm that this fixes our case.

Revision history for this message
John A Meinel (jameinel) wrote :

Landed across the 2.x series (2.0.7, 2.1.?, 2.2.?, 2.3.1, 2.4b1)

Changed in bzr:
status: In Progress → Triaged
status: Triaged → Fix Released
milestone: none → 2.4b1
Revision history for this message
Vincent Ladeuil (vila) wrote :

That's 2.1.4, 2.2.5 as presented in the milestone menu ;) Better keep track of them *now*

Revision history for this message
Chris Coulson (chrisccoulson) wrote :

I've just reported what looks like a similar issue to this (bug 752919, although the retracer marked it as a duplicate of what looks like a different bug). It just started happening tonight on a branch that I've been happily pushing to for months already

Revision history for this message
John A Meinel (jameinel) wrote :

Marking this as a bug against Launchpad, because they need to upgrade to either bzr-2.3.1 or bzr-2.2.5 to get this fix. Until then, people pushing to Launchpad will run into this, because the autopack is being done server-side. (eg bug #752919)

Deryck Hodge (deryck)
Changed in launchpad:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Robert Collins (lifeless) wrote :

We are running 2.5mumble now.

no longer affects: launchpad/2.1
no longer affects: launchpad/2.2
Changed in launchpad:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.