BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps

Bug #806348 reported by Andrew Bennetts
72
This bug affects 12 people
Affects Status Importance Assigned to Milestone
Bazaar
Confirmed
High
Unassigned
Breezy
Triaged
High
Unassigned
Launchpad itself
Triaged
Critical
Unassigned
Ubuntu Distributed Development
Confirmed
High
Unassigned

Bug Description

Currently two packages are failing to import with this error:

http://package-import.ubuntu.com/status/libffi.html
http://package-import.ubuntu.com/status/compiz-fusion-plugins-extra.html

[Current list of all failures due to that error: http://package-import.ubuntu.com/status/6fce5df2a9d46a7fd3e343fa5bc74151.html]

The problem appears to be that a very particular set of circumstances can cause fetching from a stacked repository via a smart server to omit the CHK records for a fetched inventory. Rather than trying describe the intricacies here I suggest reading the failing test added in the linked branch.

Traceback (most recent call last):
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 1129, in <module>
    persistent_download_cache=options.persistent_download_cache))
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 1035, in main
    possible_transports=possible_transports)
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 752, in create_update_branches
    bstore, possible_transports=possible_transports)
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 278, in create_updates_branch
    rev_id, possible_transports=[to_transport])
  File "/var/lib/python-support/python2.5/bzrlib/controldir.py", line 374, in sprout
    create_tree_if_local=create_tree_if_local)
  File "/var/lib/python-support/python2.5/bzrlib/cleanup.py", line 131, in run
    self.cleanups, self.func, self, *args, **kwargs)
  File "/var/lib/python-support/python2.5/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/var/lib/python-support/python2.5/bzrlib/controldir.py", line 425, in _sprout
    result_repo.fetch(source_repository, revision_id=revision_id)
  File "/var/lib/python-support/python2.5/bzrlib/repository.py", line 1781, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "<string>", line 4, in fetch_write_locked
  File "/var/lib/python-support/python2.5/bzrlib/repository.py", line 3437, in fetch
    find_ghosts=find_ghosts)
  File "/var/lib/python-support/python2.5/bzrlib/fetch.py", line 73, in __init__
    self.__fetch()
  File "/var/lib/python-support/python2.5/bzrlib/fetch.py", line 99, in __fetch
    self._fetch_everything_for_search(search)
  File "/var/lib/python-support/python2.5/bzrlib/fetch.py", line 127, in _fetch_everything_for_search
    stream, from_format, [])
  File "/var/lib/python-support/python2.5/bzrlib/repository.py", line 4105, in insert_stream
    hint = self.target_repo.commit_write_group()
  File "/var/lib/python-support/python2.5/bzrlib/repository.py", line 1640, in commit_write_group
    result = self._commit_write_group()
  File "/var/lib/python-support/python2.5/bzrlib/repofmt/pack_repo.py", line 2333, in _commit_write_group
    hint = self._pack_collection._commit_write_group()
  File "/var/lib/python-support/python2.5/bzrlib/repofmt/pack_repo.py", line 2171, in _commit_write_group
    "Cannot add revision(s) to repository: " + problems_summary)
bzrlib.errors.BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps
missing text keys: [StaticTuple('alpha-20090627035649-17514uafp7utrch0-176', '<email address hidden>'), StaticTuple('arm-20090627035649-17514uafp7utrch0-189', '<email address hidden>'), StaticTuple('avr32-20091012021018-0qbucxo52gmyrlgv-1', '<email address hidden>'), StaticTuple('config-20090627035649-17514uafp7utrch0-23', '<email address hidden>'), StaticTuple('cris-20090627035649-17514uafp7utrch0-198', '<email address hidden>'), StaticTuple('doc-20090627035649-17514uafp7utrch0-12', '<email address hidden>'), StaticTuple('ffi.c-20091012021018-0qbucxo52gmyrlgv-4', '<email address hidden>'), StaticTuple('ffitarget.h-20091012021018-0qbucxo52gmyrlgv-3', '<email address hidden>'), StaticTuple('frv-20090627035649-17514uafp7utrch0-206', '<email address hidden>'), StaticTuple('ia64-20090627035649-17514uafp7utrch0-193', '<email address hidden>'), StaticTuple('include-20090627035649-17514uafp7utrch0-249', '<email address hidden>'), StaticTuple('lib-20090627035649-17514uafp7utrch0-26', '<email address hidden>'), StaticTuple('libffi.call-20090627035649-17514uafp7utrch0-35', '<email address hidden>'), StaticTuple('libffi.pc.in-20090627035649-17514uafp7utrch0-6', '<email address hidden>'), StaticTuple('libffi.special-20090627035649-17514uafp7utrch0-30', '<email address hidden>'), StaticTuple('m32r-20090627035649-17514uafp7utrch0-226', '<email address hidden>'), StaticTuple('m68k-20090627035649-17514uafp7utrch0-202', '<email address hidden>'), StaticTuple('man-20090627035649-17514uafp7utrch0-242', '<email address hidden>'), StaticTuple('mips-20090627035649-17514uafp7utrch0-180', '<email address hidden>'), StaticTuple('pa-20090627035649-17514uafp7utrch0-144', '<email address hidden>'), StaticTuple('powerpc-20090627035649-17514uafp7utrch0-163', '<email address hidden>'), StaticTuple('s390-20090627035649-17514uafp7utrch0-222', '<email address hidden>'), StaticTuple('sh-20090627035649-17514uafp7utrch0-185', '<email address hidden>'), StaticTuple('sh64-20090627035649-17514uafp7utrch0-157', '<email address hidden>'), StaticTuple('sparc-20090627035649-17514uafp7utrch0-150', '<email address hidden>'), StaticTuple('src-20090627035649-17514uafp7utrch0-143', '<email address hidden>'), StaticTuple('sysv.s-20091012021018-0qbucxo52gmyrlgv-2', '<email address hidden>'), StaticTuple('targetlibpath.exp-20090627035649-17514uafp7utrch0-27', '<email address hidden>'), StaticTuple('testsuite-20090627035649-17514uafp7utrch0-22', '<email address hidden>'), StaticTuple('texinfo.tex-20090627035649-17514uafp7utrch0-239', '<email address hidden>'), StaticTuple('tree_root-20090627035649-17514uafp7utrch0-2', '<email address hidden>'), StaticTuple('wrapper.exp-20090627035649-17514uafp7utrch0-28', '<email address hidden>'), StaticTuple('x86-20090627035649-17514uafp7utrch0-211', '<email address hidden>')]

Revision history for this message
Andrew Bennetts (spiv) wrote :

I wouldn't be shocked to find that some other failures are caused or at least related, e.g. <https://bugs.launchpad.net/bzr/+bug/772935>

Martin Pool (mbp)
Changed in bzr:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Diogo Matsubara (matsubara) wrote :

I added a bug task for LP due to OOPS-2021BZR207511 which looks like a similar error. Could someone confirm this is the same or if it requires a new bug?

Changed in launchpad:
status: New → Triaged
importance: Undecided → Critical
tags: added: oops
Revision history for this message
Martin Pool (mbp) wrote :

@matsubara unfortunately for some reason there's no actual traceback but it looks similar.

Revision history for this message
Martin Pool (mbp) wrote :

https://bugs.launchpad.net/launchpad/+bug/812106 for the lack of a traceback in the oops.

Revision history for this message
Andrew Bennetts (spiv) wrote :

The linked branch, lp:~spiv/bzr/stacked-fetch-same-chks, adds a failing test for this case (or at least a case *like* this). The challenge is figuring out what the fix should be.

This is a pretty fiddly problem, but I'll try to explain it as clearly and concisely as possible. When fetching multiple heads from a stacked repository via a smart server the assumptions we use to stream only O(changes) data rather O(tree) data aren't necessarily correct. The assumptions is basically “we can omit transferring records that are unchanged in the parent revisions of the set we are sending”, and the justification is that the parent will be present anyway: we assume another fetch in the chain of stacked fetches will include it (or that the target has already got that fetch anyway).

The problem is that justification isn't true: the other fetches in the stacked chain, following the same assumption, can exclude the same CHK records, so it is possible they are never sent. It is possible to construct fetches that always have ‘present parent inventories’ at every step in the stacking chain, by starting with a fetch with multiple heads. e.g. a revision graph like:
 * D has parents: C, A
 * C has parents: B
 * B has parents: A
 * A has parents: (none)
And then fetch [D,A] from the one repo (with C as present parent inventory), and then [C,B] from the next (and A is present, so is present parent inventory). If A and C share some records, both fetches will conclude they can omit those records. That then triggers BzrCheckError during commit_write_group (thank goodness for that integrity check!).

Examples of records that can be omitted by this erroneous logic:
  * the parent_id_basename_to_file_id map can easily be shared between multiple revisions ('bzr revert -r -2 && bzr commit'), which shares the CHK root key
  * an id_to_entry map with some LeafNodes in common (e.g. it's possible to have all revisions sharing a LeafNode with just the tree root, and have that leaf be the same in all revisions).
  * text to sends are determined by the id_to_entry records we send, so if any id_to_entry nodes are missing, then so are some texts.

The challenge is how we can fix this while still sending only O(changes) data (which is a key benefit of using stacking). What doesn't work:
  * simply re-requesting those inventories in the 'fix missing inventories' step, because that doesn't cause texts to be transferred (perhaps if we allowed a *third* fetch round to fixup those missing texts then maybe that would work? Ugh.)
  * discarding known-interesting CHK roots from the uninteresting-roots set before calling iter_interesting_nodes: that still allows shared non-root CHK nodes to be omitted.
  * dropping the filtering of “uninteresting” CHK nodes: that would regress to sending O(tree)-sized data.

Ideas welcome!

Revision history for this message
Andrew Bennetts (spiv) wrote :

I should note that another aspect here is that so far it appears that this situation only occurs when fetching multiple heads (as now commonly happens because we fetch missing tags in recent bzr versions). I'm not certain that fetching one head at a time can't hit a similar problem, but if that is safe it suggests that perhaps a fix could be:
 * when calculating which records to stream do so independently for each head and then send the union of those streams.

That feels too expensive to me though (both in computation effort, and in possibly sending redundant data).

Revision history for this message
Martin Pool (mbp) wrote :

I don't know what's changed but this seems to be happening to an increasing number of branches in <http://package-import.ubuntu.com/status/>, so I'm going to escalate this in bzr.

Changed in bzr:
importance: High → Critical
Revision history for this message
Andrew Bennetts (spiv) wrote : Re: [Bug 806348] Re: BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps

Martin Pool wrote:
> I don't know what's changed but this seems to be happening to an
> increasing number of branches in <http://package-
> import.ubuntu.com/status/>, so I'm going to escalate this in bzr.

The hypothesis is that when jubany upgraded to lucid it also upgraded to
a version of bzr that fetches tags by default… and fetching multiple
tips simultaneously seems to be a key (or maybe just common?) trigger
for this edge case.

Revision history for this message
Max Bowsher (maxb) wrote :

I just requeued all of the following packages now that we're on bzr 2.3.4 (instead of 2.4b4):

anjuta bin-prot binfmt-support binutils binutils-z80 bird c++-annotations cacti cheese cl-sql compiz-fusion-plugins-extra conkeror crash cupt d-shlibs dbus debhelper dh-make-perl dialog dkms dma doxygen dpkg-cross feh fglrx-driver firebug firmware-nonfree fizmo freebsd-libs freetalk gawk gecode giplet gitpkg glpi gmerlin gnome-icon-theme gnumed-client gnumed-server gnupg-pkcs11-scd gource gpodder gpsd gpsk31 gsmartcontrol gst0.10-python gstreamer0.10-ffmpeg gtest gtk+2.0 gtkwave hardening-wrapper hatari hdf5 heimdal hplip hugin iceweasel imagej iptables iso-codes jags javatools jqueryui kdeaccessibility kdeadmin kdegames kdegraphics kdeplasma-addons kdewebdev klibc ldb libaqbanking libcompress-raw-zlib-perl libconfig-model-perl libdatetime-timezone-perl libdevel-declare-perl libdist-zilla-perl libffi libgadu libhtml-template-pro-perl libimager-perl libio-async-perl libisoburn libisofs libkarma liblocale-gettext-perl libmail-box-perl libmodule-corelist-perl libmoose-perl libnet-ssh2-perl libpng libsvn-hooks-perl libtest-www-mechanize-perl libtorrent-rasterbar libusb-1.0 linux-2.6 liquidsoap live-manual lua5.1 massxpert merkaartor meta-kde metacity miro mlt mozilla-devscripts mozilla-noscript nbd neon27 nginx nilfs-tools nodm nss openmsx-debugger org-mode pastebinit pbzip2 pciutils pidgin pilot-link pkg-kde-tools pstoedit puppet pyparted python-application python-networkx pywebdav qlandkartegt qsynth r-base r-zoo rcpp rxvt-unicode sane-backends serf sgt-puzzles shorewall-doc socat sphinx sphinxsearch squid3 sugar-hulahop swt-gtk tangerine thunderbird tinymce tomcat6 torbutton tortoisehg trac tryton-client tryton-modules-account tryton-modules-account-de-skr03 tryton-modules-account-statement tryton-modules-analytic-invoice tryton-modules-country tryton-modules-purchase tryton-modules-sale tryton-modules-stock tryton-modules-stock-location-sequence tryton-server tzdata unbound usermode valgrind varnish viking vinagre vlc w3m wdm webgui websvn wireshark wvdial xml-security-c xserver-xorg-input-mouse zendframework zsh-beta

Revision history for this message
Martin Pool (mbp) wrote :

Moving jubany back to bzr 2.3 did seem to avoid these problems, so fixing bug 771184 will probably largely fix this.

Not having versioned tags makes it very hard to get rid of tags; therefore very hard to get rid of tags that point at irrelevant history; so that keeps floating around.

Revision history for this message
John A Meinel (jameinel) wrote :

As a quick regression avoidance, bug #771184 is relevant.

Changed in udd:
assignee: Andrew Bennetts (spiv) → nobody
Vincent Ladeuil (vila)
Changed in udd:
status: In Progress → Confirmed
Changed in bzr:
importance: Critical → High
John A Meinel (jameinel)
Changed in bzr:
assignee: nobody → John A Meinel (jameinel)
Martin Packman (gz)
tags: added: affects-twisted
Revision history for this message
Martin Packman (gz) wrote :

Is there any practical difference between this bug 485601?

Revision history for this message
Jelmer Vernooij (jelmer) wrote :

bug 485601 is about bzr-svn triggering this bug - creating slightly inconsistent text revision metadata. That issue has been fixed.

Jelmer Vernooij (jelmer)
Changed in bzr:
assignee: John A Meinel (jameinel) → Jelmer Vernooij (jelmer)
Jelmer Vernooij (jelmer)
Changed in bzr:
status: Confirmed → In Progress
Jelmer Vernooij (jelmer)
Changed in bzr:
assignee: Jelmer Vernooij (jelmer) → nobody
status: In Progress → Triaged
status: Triaged → Confirmed
William Grant (wgrant)
tags: added: bzr
Revision history for this message
Andrew McDonnell (andymc73) wrote :

I hit this just now attempting to checkout lp:maria
Although my stack trace more resembles that in one of the duplicate bugs - bug 890373
(My stack trace is effectively the same as that bug, with some very minor differences in lines numbers)

It also deleted everything. It would have been useful if it left the downloaded repo at the most recent version it got to so it was possible to attempt to resume... This after about ~400 MBytes and 1/2 hour ... X-(

bzr 2.5.1 on python 2.6.6 on squeeze

Revision history for this message
Joni-Pekka Kurronen (joni-kurronen) wrote :
Download full text (3.3 KiB)

joni@mpi1:/mpi3/S3/maria$ bzr branch lp:maria trunk
bzr: ERROR: bzrlib.errors.BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps
missing chk node(s) for parent_id_basename_to_file_id maps

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 920, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 1131, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 673, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 695, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 136, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 166, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/builtins.py", line 1493, in run
    source_branch=br_from)
  File "/usr/lib/python2.7/dist-packages/bzrlib/bzrdir.py", line 366, in sprout
    create_tree_if_local=create_tree_if_local)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 132, in run
    self.cleanups, self.func, self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 166, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/bzrdir.py", line 416, in _sprout
    result_repo.fetch(source_repository, fetch_spec=fetch_spec)
  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 1267, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/usr/lib/python2.7/dist-packages/bzrlib/decorators.py", line 218, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 2584, in fetch
    find_ghosts=find_ghosts)
  File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 77, in __init__
    self.__fetch()
  File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 104, in __fetch
    self._fetch_everything_for_search(search_result)
  File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 132, in _fetch_everything_for_search
    stream, from_format, [])
  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 2050, in insert_stream
    hint = self.target_repo.commit_write_group()
  File "/usr/lib/python2.7/dist-packages/bzrlib/repository.py", line 633, in commit_write_group
    result = self._commit_write_group()
  File "/usr/lib/python2.7/dist-packages/bzrlib/repofmt/pack_repo.py", line 1727, in _commit_write_group
    hint = self._pack_collection._commit_write_group()
  File "/usr/lib/python2.7/dist-packages/bzrlib/repofmt/pack_repo.py", line 1607, in _commit_write_group
    "Cannot add revision(s) to repository: " + problems_summary)
BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps
missing chk node(s) for ...

Read more...

Jelmer Vernooij (jelmer)
tags: added: check-for-breezy
Jelmer Vernooij (jelmer)
tags: added: data-integrity
removed: check-for-breezy
Changed in brz:
status: New → Triaged
importance: Undecided → High
Jelmer Vernooij (jelmer)
tags: added: bzr-format
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.