BzrCheckError: Cannot add revision(s) to repository: missing referenced chk root keys

Bug #522637 reported by Husen Daudi on 2010-02-16
124
This bug affects 21 people
Affects Status Importance Assigned to Milestone
Bazaar
High
Andrew Bennetts
2.0
High
Unassigned
2.1
Undecided
Unassigned
Do
Undecided
Unassigned
Launchpad itself
Low
Unassigned

Bug Description

Summary
----------
Some versions of bzr could introduce non-canonical CHK maps into a repository which can cause tracebacks when fetching from that repository. There is no data loss.

Repair
---------
There's a repair tool in bzr trunk (and will be in bzr 2.3):

 - get a copy of bzr trunk: "bzr branch lp:bzr"
 - run "bzr reconcile --canonicalize-chks AFFECTED_REPO"

Original description
----------

Hello all,
after updating my bzr version from 1.6 to 2.0.4 everytime when I do any bzr operation it says
Doing on-the-fly conversion from <RepositoryFormatKnitPack4> to (remote).
This may take some time. Upgrade the repositories to the same format for better performance.

so i change format to 2a and now i can no do any operation on my branch.

here is the trace back
PythonVersion: 2.5.2
Traceback:
 Traceback (most recent call last):
   File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 842, in exception_to_return_code
     return the_callable(*args, **kwargs)
   File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 1037, in run_bzr
     ret = run(*run_argv)
   File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 654, in run_argv_aliases
     return self.run(**all_cmd_args)
   File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 1017, in run
     possible_transports=possible_transports, local=local)
   File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.5/site-packages/bzrlib/workingtree.py", line 1611, in pull
     local=local)
   File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 948, in pull
     possible_transports=possible_transports, *args, **kwargs)
   File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 3187, in pull
     _override_hook_target=_override_hook_target)
   File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 3064, in pull
     overwrite=overwrite, graph=graph)
   File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 896, in update_revisions
     overwrite, graph)
   File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 3007, in update_revisions
     self.target.fetch(self.source, stop_revision)
   File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 579, in fetch
     pb=pb)
   File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 1696, in fetch
     find_ghosts=find_ghosts, fetch_spec=fetch_spec)
   File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 3414, in fetch
     pb=pb, find_ghosts=find_ghosts)
   File "/usr/lib/python2.5/site-packages/bzrlib/fetch.py", line 81, in __init__
     self.__fetch()
   File "/usr/lib/python2.5/site-packages/bzrlib/fetch.py", line 107, in __fetch
     self._fetch_everything_for_search(search)
   File "/usr/lib/python2.5/site-packages/bzrlib/fetch.py", line 135, in _fetch_everything_for_search
     stream, from_format, [])
   File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 4245, in insert_stream
     return self._locked_insert_stream(stream, src_format, is_resume)
   File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 4341, in _locked_insert_stream
     hint = self.target_repo.commit_write_group()
   File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 1555, in commit_write_group
     result = self._commit_write_group()
   File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 2273, in _commit_write_group
     hint = self._pack_collection._commit_write_group()
   File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 2105, in _commit_write_group
     "Cannot add revision(s) to repository: " + problems_summary)
 BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing referenced chk root keys: [('sha1:fd73461fbfd0267489266f319c50f23c794efcf3',)]

UserEncoding: UTF-8
UserGroups: adm admin cdrom dialout lpadmin plugdev sambashare

Related branches

Martin Pool (mbp) on 2010-02-17
summary: - not able to pull after bzr upgrade --2a
+ BzrCheckError: Cannot add revision(s) to repository: missing referenced
+ chk root keys
Changed in bzr:
importance: Undecided → High
status: New → Confirmed

Hi

I have the same problem with this repository https://code.launchpad.net/~openerp/openobject-server/5.0

It recently update to format 2a, in my local copy i launch "bzr check" and "bzr upgrade" in 2.0.4
and today a bzr pull give me an error (check the logfile), i just upgrade to 2.1.0 and have the same problem.

Regards,

Joke de Buhr (joke) wrote :

Pulling from bzr+ssh://bazaar.launchpad.net/~openerp/openobject-server/5.0/ into my branch (in a repository) I get an other missing sha1 error: missing referenced chk root keys: [StaticTuple('sha1:0c211a78d10233c47ce084b5b2ecbf44c1d09d40',)]

Creating a new branch within the same repository doesn't work either. Creating a completely new branch from https://code.launchpad.net/~openerp/openobject-server/5.0 is successful though.

Martin Pool (mbp) on 2010-03-17
tags: added: 2a
Robert Collins (lifeless) wrote :

So, chk root keys are pointed at by inventories, this is an unusual error to get. To debug I think we should dump the references revisions, inventories, referenced chk roots, and see if a pattern emerges.

I suspect the old branch was missing content and thus it wasn't migrated fully.

Martin Pool (mbp) wrote :

bug 539052 is a similar occurrence of this, also in the openerp branches. I wonder if they were migrated from bzr-svn or converted using an early version of bzr's 2a support, because similar problems seem to have occurred there.

Johannes Sasongko (sjohannes) wrote :

Just a note that I'm recently getting this problem from lp:exaile. I ran the upgrade locally (bzr upgrade lp:exaile) because I didn't notice the "upgrade branch" link, using bzr 2.1.1. My local branch was on a shared repository, if that's relevant. The remote branch was migrated from svn a long time ago, but I don't know using which tool.

I've thoughtlessly deleted my old repository and can't restore it from a backup, but I've pointed users here if they are experiencing the problem and can help with testing stuff.

Karoly Negyesi (karoly) wrote :

Here is a crash log of an occurence of this bug.

Numérigraphe (numerigraphe) wrote :

I've encountered a similar problem, again with OpenERP server 5.0 ; here is as much detail as I could gather.

I use a local repo to host 3 branches : "5.0", "dev" and "prod".
- "5.0" is a checkout of lp:~openerp/openobject-server/5.0/
- "dev" is where I make my daily work
- "prod" is a checkout of a distant (bzr+ssh) standalone branch called "server", on the production host.
Often when a release is tagged in "5.0", I'll merge it into "dev".
Eventually I'll merge "dev" into "prod", and "bzr update" the distant "server" branch via SSH.
That worked fine for the tag "5.0.7" and the tags before.

Now I merged the tag "5.0.10" from "5.0" into "dev", but when I merged "dev" into "prod" I couldn't commit (see crash file attached)
:
ErrorFromSmartServer: Error received from smart server: ('error', "Internal check failed: Cannot add revision(s) to repository: missing referenced chk root keys: [StaticTuple('sha1:0deb2b3b15a16211b3addbfb524efe48534b139c',)]")
Commit --local worked but then push failed, and uncommit failed too.
"bzr check" on the repo and the distant branch are OK.

I tried to rebuild a new repo and import my branches into it, but met the same error when I pushed to the distant branch.

I rsynced a local copy of the distant branch to make sure it's not a bzr+ssh issue, and met the same error whe I tried to push to it.

Via SSH, I created a new distant branch from the old distant branch and tried to push to it, and met the same error.

Finally I created a new repo, imported my branches into it, then created a new standalone branch from my new "prod" branch and rsync'ed it to the distant host to replace the "server" branch. That seems to work all right now.

I have a backup copy of the initial repo and the old distant branch, I'll make them available if someone wants to investigate.

Lionel.

Martin Pool (mbp) on 2010-06-07
Changed in bzr:
milestone: none → 2.2.0
Cristian Salamea (ovnicraft) wrote :

Hi folks, i think the bug is priority in a 2.1.x release can be pointed in them?

Regards,

I get the same type of error when I try to do: bzr branch lp:do - actually I already had a branch, which I migrated as requested. Then I couldn't pull anymore, so I deleted the directory and tried to branch again, which doesn't work. However, if I comment launchpad_username in ~/.bazaar/bazaar.conf and the whole [Launchpad] section in ~/.bazaar/authentification.conf, then I can branch without any problem.

I attached the crash file obtained when running "bzr branch lp:do" with my LP/SSH credentials specified.

Andrew Bennetts (spiv) wrote :

Nicolas: thanks for that. I can reproduce this with "bzr branch lp:do" too! I'm grabbing a copy of lp:do's files (and the files of its stacked-on branch) now so that I can investigate in more detail.

Andrew Bennetts (spiv) wrote :

I've started actively working on this. Unsurprisingly it looks like a bad interaction with stacking.

Specifically for the lp:do example, there are multiple inventories in the stream from the stacked-on branch with a parent_id_basename_to_file_id root key of sha1:576fa262710cb8cadf998f50c33f4fa28ac6f57b, but that key is apparently being filtered out due to also being the parent_id_basename_to_file_id root key of an inventory at the edge of ancestry.

Changed in bzr:
assignee: nobody → Andrew Bennetts (spiv)
status: Confirmed → In Progress
tags: added: stacking
Andrew Bennetts (spiv) wrote :

Short answer: the repository has bad data, but it can be repaired. I have a fix for the cause of the bad data (see branch linked to this bug) and will provide a tool to do the repair soon. There is no data loss.

Long, jargon-filled answer:

The root cause here is that CHKMap.apply_delta and CHKMap._check_remap had bugs that meant that it could generate a CHK map with a non-canonical form. In the case of lp:do, there are some inventories records that exist in both lp:do and the branch it is stacked on (lp:do/0.9), but that have different CHK roots. Mostly this sort of inconsistency doesn't cause trouble, but in this case one of the inventories is at the stacking boundary, and fetch relies on the invariant that a given inventory record will be identical in both repositories: it fetches the inventory record for a parent inventory at the stacking boundary from one repo, then fetches CHK roots from the other. If the source repositories disagree on the CHK roots for that inventory then the target finds the CHK roots are missing, causing the error reported in this bug.

Aaron Bentley (abentley) wrote :

Submitting merge directives with bundles to <email address hidden> is causing this to appear in the oops: OOPS-1663CMP3

tags: added: code-review
Aaron Bentley (abentley) on 2010-07-22
Changed in launchpad-code:
status: New → Triaged
importance: Undecided → Low
Nebu Pookins (nebupookins) wrote :

Can't do a checkout of lp:do because of this bug.

Martin Pool (mbp) wrote :

Will be fixed in 2.0.6, in the next 2.1, and it 2.2.0. As a separate task we need a thing to fix repositories affected by this: bug 613698.

Alex Launi (alexlauni) on 2010-08-05
Changed in do:
status: New → Invalid
John A Meinel (jameinel) on 2010-08-06
Changed in bzr:
status: In Progress → Fix Released
Andrew Bennetts (spiv) wrote :

The root cause has been fixed, and as per bug 613698 there's now a repair tool available in trunk.

If you have an affected repository, get a copy of bzr trunk (bzr branch lp:bzr), and run "bzr reconcile --canonicalize-chks" on the affected repository.

(I'll copy that instruction into the description of this bug to help people find it.)

Andrew Bennetts (spiv) on 2010-08-16
description: updated
description: updated
Vincent Ladeuil (vila) on 2010-09-30
Changed in bzr:
milestone: 2.2.0 → 2.2.1

Hello everyone,

Seeing that the repair tool has now made it into trunk with bug 613698, I take it Launchpad admins should run it on affected repositories on the Launchpad repositories, right? I don't think I can do it remotely, can I?

Should I open another bug report for it?
Seeing that current one is still Triaged for launchpad-code, I would like to request this for the official OpenObject/OpenERP branches:
   - openobject-server/trunk and openobject-server/5.0
   - openobject-addons/trunk and openobject-addons/5.0
   - openobject-client/trunk and openobject-client/5.0
   - openobject-client-web/trunk and openobject-client-web/5.0

I'm not sure all branches are affected, but I don't know whether I can check that easily. None of them are stacked, however they do have hundreds of dependent stacked branches, so please make sure a fresh backup is available before applying :-)

Thanks a lot!

Tim Penhey (thumper) wrote :

Olivier I think you should be able to run it remotely, at least I can't see a reason why you couldn't.

Aaron Bentley (abentley) wrote :

Launchpad 10.12 uses bzr 2.2.2

Changed in launchpad-code:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Related questions