Bug #485601 “missing chk node(s) for id_to_entry maps” : Bugs : Bazaar

Geoff Jacobsen (geoffj) wrote on 2009-11-20:

#1

crash report Edit (43.4 KiB, text/plain)

John A Meinel (jameinel) wrote on 2009-11-20: Re: [Bug 485601] [NEW] missing chk node(s) for id_to_entry maps

#2

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Geoff Jacobsen wrote:
> Public bug reported:
>
> This occurs in conjunction with the bzr-svn plugin (version 1.0.0dev)
> and bzr versions 2.0.0 and 2.0.1
>
> I have an original bzr repository which is push into a subversion
> repository. This is done half hourly if there are any changes on the
> original branch. No changes are done on the svn repo other than the
> push.
>
> Eventually there is a branch and merge on the original which results not
> being able to push to svn since a rebase is needed.
>
> At this stage if I "bzr branch" the svn repo and try to pull the
> original into it I get the error (both on Ubuntu and Solaris) also if I
> try a rebase on a "bzr co svn-repo" I get the same error.
>
> ** Affects: bzr
> Importance: Undecided
> Status: New
>
>
> ** Tags: bzr-svn
>

affects bzr-svn

I don't know that it is bzr-svn's fault, but as it is provoking this
situation, I'm including bzr-svn in the bug report.

affects bzr
status confirmed
importance medium

My guess is that there is a bad interaction with ghosts introduced by
bzr-svn that are(n't) getting filled in when fetching from the original
bzr branch.

Just shooting out a possibility, I would guess that you end up with a
revision graph like:

A
|\
B C
|/|
D | # pushed to svn
|\|
E F
|/
G

And fetching from SVN gives you something like:

A
|
B ? #ghost
|/
D

Then fetching from the bzr branch tries to fill in only new children,
such as:

A
|
B ? #still a ghost
|/|
D |
|\|
E F
|/
G

And at that point, we fall over. The stream request did not request the
chk pages for C, so the server specifically excludes them.

However, those pages end up also present in F (and not present in D), so
we don't have them locally.

The various failures:

1) The ancestry search sees that we have D, and stops searching. Thus we
never see that we are missing revision "C" locally.

2) The fetch requests revision E F G versus revision D.

3) The remote side translates that into E F G versus C D.

I haven't proven any of this. But basically it is a failure because
bzr-svn introduces a ghost which we then don't fetch. And the current
codebase expects that if you have a revision, you have all of its
ancestry. (Which ghosts violate.) And the chk map code is especially
sensitive to this.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksGucEACgkQJdeBCYSNAAODuACfaKJ085K7Y3dgWrPTZq4vZnf5
UN0AnjVtFsbMON1N0fw9ghHQ+EBpfaJD
=hSov
-----END PGP SIGNATURE-----

Changed in bzr:
importance:	Undecided → Medium
status:	New → Confirmed

Jelmer Vernooij (jelmer) wrote on 2010-04-07:

#3

Geoff, is your repository public?

Geoff Jacobsen (geoffj) wrote on 2010-04-07: Re: [Bug 485601] Re: missing chk node(s) for id_to_entry maps

#4

On 8 April 2010 11:24, Jelmer Vernooij <email address hidden> wrote:

> Geoff, is your repository public?
>
>
Sorry, no but the problem is easy to re-produce. All you need to do is push
a change to subversion which will cause a re-org of the tree.

Jelmer Vernooij (jelmer) wrote on 2010-04-08:

#5

On Wed, 2010-04-07 at 23:59 +0000, Geoff Jacobsen wrote:
> On 8 April 2010 11:24, Jelmer Vernooij <email address hidden> wrote:
>
> > Geoff, is your repository public?
> Sorry, no but the problem is easy to re-produce. All you need to do is push
> a change to subversion which will cause a re-org of the tree.
I haven't encountered this problem before and I've pushed quite a few
repository reorganizations using bzr-svn. It's definitely not any
reorganization that triggers this bug but something more specific.

status incomplete

Cheers,

Jelmer

Jelmer Vernooij (jelmer) on 2010-04-08

Changed in bzr:
status:	Confirmed → Incomplete

Geoff Jacobsen (geoffj) wrote on 2010-04-08:

#6

I'll try and publish the steps to create it on an ubuntu box but I am tied
up for the next week

On 8 April 2010 12:30, Jelmer Vernooij <email address hidden> wrote:

> ** Changed in: bzr
> Status: Confirmed => Incomplete
>
> --
> missing chk node(s) for id_to_entry maps
> https://bugs.launchpad.net/bugs/485601
> You received this bug notification because you are a direct subscriber
> of the bug.
>

Philip Peitsch (philip-peitsch) wrote on 2010-05-05:

#7

I have suffered the same problem in an annoyingly unreproducible way. I have tried many combinations of merging, pushing and pulling between bzr branches of the same svn repo project, and cant find a reproducible way.

Geoff... were you using any shared bzr repositories when you suffered this problem?

Geoff Jacobsen (geoffj) wrote on 2010-05-05:

#8

Yes the repos are shared and it happens when two merges
occur independently and then merged to parent forcing one tree to be
reordered.

On 5 May 2010 17:29, Philip Peitsch <email address hidden> wrote:

> I have suffered the same problem in an annoyingly unreproducible way. I
> have tried many combinations of merging, pushing and pulling between bzr
> branches of the same svn repo project, and cant find a reproducible way.
>
> Geoff... were you using any shared bzr repositories when you suffered
> this problem?
>
> --
> missing chk node(s) for id_to_entry maps
> https://bugs.launchpad.net/bugs/485601
> You received this bug notification because you are a direct subscriber
> of the bug.
>

Philip Peitsch (philip-peitsch) wrote on 2010-05-05:

#9

That's what I am experiencing as well. I have two developers with independant branches of the same svn code in two separate shared repositories with no working trees, and if one dev tries to bzr pull from the other dev's branch, this crash is always experienced.

Is there a way to get more information out about the branch history etc to help with debugging this problem? I unfortunately can't simply release the source as it is a propriatary app (and around 2Gb worth of histroy at this point I might add), but I am happy to summarize most data and send to specific emails if that would be able to provide any extra information.

jeffreycameron@gmail.com (jeffreycameron) wrote on 2010-05-19:

#10

We're experiencing this problem as well ... any solution in the pipeline?

Philip Peitsch (philip-peitsch) wrote on 2010-05-30:

#11

This bug has pretty much sunk all our bzr branches now (we are/were trying to migrate from svn).

Is there likely to be any time soon to get some help looking at this? Though I can't open up our source control access to outside access.... I can make myself available on IRC between 9-5pm Melbourne, Australia time if that's helpful.

Philip Peitsch (philip-peitsch) wrote on 2010-06-01:

#12

Geoff & jeffrey... I am wondering if this is possibly caused by Bug #587819. Can you both see if you're able to create clean checkouts of the svn repo using a non-shared-repository branch? (i.e., outside the shared repo, do a bzr co <svn repo url>). If you had the race condition bug, it should fail in theory...

Geoff Jacobsen (geoffj) wrote on 2010-06-01:

#13

I am the only one accessing the svn repository and the bug is
100% reproducible so I don't believe it has to do with bug #587819

Philip Peitsch (philip-peitsch) wrote on 2010-06-01:

#14

Ahh. Fair enough. How many revisions do you have in your branch at the moment if I may ask? I'm interested in fixing this one, but am just about to lose my test repo due to the previously mentioned issue. Do you have any idea on reproduction steps yet?

Thanks,

Philip

Jelmer Vernooij (jelmer) wrote on 2010-06-07:

#15

Marking as incomplete as I haven't encountered this myself and I don't have a way to reproduce this.

Changed in bzr-svn:
status:	New → Incomplete

Arnaud Jeansen (ajeans) wrote on 2010-06-23:

#16

Download full text (3.6 KiB)

I just had this problem, here are the steps that brought this error message. I will try to reproduce later with a public svn repository. Here are the logs to help me reproduce it later.

Note that we use subversion in a braindead manner (i.e. we commit inside the tags).

Subversion side :
We have tagged twice our trunk, first with tag "r3", then with tag "r4". Then, we did several commits in both tags, both from svn and from bzr (by pulling changes then committing).

Bazaar side:
1) Branch from the subversion tag r3
repo$ bzr branch svn://host/repo/tags/r3
2) Branch from the subversion tag r4
repo$ bzr branch svn://host/repo/tags/r4
3) Try to merge or pull a revision from the bzr branch "r3" into the branch "r4"
repo/r4$ bzr merge -r1218 ../r3/

=> Crash with the following message

bzr: ERROR: bzrlib.errors.BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 853, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1055, in run_bzr
  File "bzrlib\commands.pyo", line 661, in run_argv_aliases
  File "bzrlib\commands.pyo", line 665, in run_direct
  File "bzrlib\cleanup.pyo", line 122, in run_simple
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "bzrlib\builtins.pyo", line 3794, in run
  File "bzrlib\builtins.pyo", line 3923, in _get_merger_from_branch
  File "bzrlib\merge.pyo", line 363, in from_revision_ids
  File "bzrlib\merge.pyo", line 505, in set_other_revision
  File "bzrlib\merge.pyo", line 522, in _maybe_fetch
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\branch.pyo", line 593, in fetch
  File "bzrlib\repository.pyo", line 1704, in fetch
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\repository.pyo", line 3432, in fetch
  File "bzrlib\fetch.pyo", line 83, in __init__
  File "bzrlib\fetch.pyo", line 109, in __fetch
  File "bzrlib\fetch.pyo", line 137, in _fetch_everything_for_search
  File "bzrlib\repository.pyo", line 4274, in insert_stream
  File "bzrlib\repository.pyo", line 4377, in _locked_insert_stream
  File "bzrlib\repository.pyo", line 1563, in commit_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 2314, in _commit_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 2152, in _commit_write_group
BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps

bzr 2.1.0 on python 2.5.4 (Windows-XP-5.1.2600-SP2)
arguments: ['c:\\Program Files\\Bazaar\\bzr.exe', 'merge', '-r1218', '../2.42r3/']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files\Bazaar\plugins\bzrtools [2.1.0]
  explorer C:\Program Files\Bazaar\plugins\explorer [1.0.0rc1]
  launchpad C:\Program Files\Bazaar\plugins\launchpad [2.1.0]
  netrc_credential_store C:\Program Files\Bazaar\plugins\netrc_credential_store [2.1.0]
  qbzr C:\Program Files\Bazaar\plugins\qbzr [0.18.1]
  rebase C:\Program Files\Bazaar\plugins\rebase [0.5.5]
  svn C:\Program Files\Bazaar\plugins\svn [1.0.2]
...

Philip Peitsch (philip-peitsch) wrote on 2010-09-30:

#17

This bug just sunk all the devs again when upgrading to 2.2.1

Philip Peitsch (philip-peitsch) wrote on 2010-09-30:

#18

Could this bug potentially be caused by https://bugs.launchpad.net/bzr/+bug/522637 ?

Jelmer Vernooij (jelmer) wrote on 2010-09-30:

#19

Unfortunately I still can't do much about this, as there is no clear way of reproducing it.

The ghost handling in bzr-svn 1.0.4 has changed significantly, so it'd be interesting to hear if anybody can still reproduce this with that version of bzr-svn and a clean clone (into a new shared repository) from their svn repository.

Philip Peitsch (philip-peitsch) wrote on 2010-10-11:

#20

This bug appears to happen using the following workflow:

* dev a & dev b both share a network repo, and both have a local repo as well
* dev a does some work and commits to their branch (this is immediately pushed up to the local repo via a bound branch)
* dev a merges this work into svn trunk
* system admin updates remote repo
* dev b updates local svn trunk VIA SVN
* dev b then attempts to merge changes from svn trunk into their local feature and commit (which is also a bound branch to the remote repo)

Explosions happen!

Andrew Bennetts (spiv) wrote on 2010-10-12:

#21

At a glance bzr-svn seems to be using bzr APIs correctly. It adds inventory objects to a bzr repo via add_inventory_by_delta, which is a public API that is intended to be used for exactly that. So that seems fine.

But I wonder if perhaps it's possible that bzr-svn isn't roundtripping bzr inventories perfectly? Is it possible that it adds a bzr revision to a SVN repo, and then another user fetches that revision and gets an inventory with the same ID but slightly different contents? That might explain this behaviour.

Andrew Bennetts (spiv) wrote on 2010-10-12:

#22

Update from IRC: we've found a revision in Philip Peitsch's scenarios that is causing this trouble. When fetched from the SVN repo into bzr repo A it has different CHK root keys than after fetching from A into B. But in both A and B the CHK maps seem to be in canonical form! So apparently A and B have different content for the same inventory?

Andrew Bennetts (spiv) wrote on 2010-10-12:

#23

Philip has sent me dumps of the CHK maps of some of the revisions converted from the same SVN repo at different times. The inventories for those revisions are the same in every detail *except* some of the inventory entries have 'revision' attributes. It appears the 'revision' attributes of inventory-entries generated by bzr-svn from SVN are changing every time a new bzr revision is added to the SVN repository!

Philip Peitsch (philip-peitsch) wrote on 2010-10-12:

#24

The original bzr repository was created after the svntrunk suffered from https://bugs.launchpad.net/bzr-svn/+bug/515850. To recover from this bug, manually deleted all bzr:XXX revision properties from the repository. I am quite sure we cleaned out all svn-caches on all PC's as well, but it's worth considering if that history has lead to this current situation somehow.

I've checked dumps of the CHK maps for the rev just prior to our first "new" bzr checkin being landed in SVN (post bug 515850), and these differ slightly. No svn checkins prior to that first bzr commit seem to differ from what I've seen however.

Philip Peitsch (philip-peitsch) wrote on 2010-10-12:

#25

Just to clarify the previous comment: To recover from this bug, manually deleted all bzr:XXX revision properties from the SUBVERSION repository, and threw away the old BZR repository. I then created a brand new checkout of svntrunk into a clean BZR repository, and this is what we've been using since then (early June, 2010)

Glyph Lefkowitz (glyph) wrote on 2010-10-28:

#26

I just reproduced this on bzr-svn 1.0.4.

I reproduced it fairly simply: I have 2 repositories, one on a desktop and one on a laptop.

desktop$ bzr init-repository .
desktop$ bzr get svn+http:.../trunk
laptop$ bzr init-repository .
laptop$ bzr get svn+http:.../trunk
laptop$ bzr branch trunk feature-branch
laptop$ hack hack hack
laptop$ bzr ci -m "stuff"
desktop$ bzr get bzr+ssh://laptop/~/feature-branch
... and that causes the traceback.

There is literally only one bzr revision in my feature branch which hasn't been pushed to SVN which seems to be causing the problem.

John Szakmeister (jszakmeister) wrote on 2010-12-23:

#27

I'm still having trouble trying to reproduce this, but I'm running into it quite frequently. I really can't take Philip's approach (I don't own the repository). :-(

Jelmer Vernooij (jelmer) wrote on 2010-12-23:

#28

On Thu, 2010-12-23 at 08:45 +0000, John Szakmeister wrote:
> I'm still having trouble trying to reproduce this, but I'm running into
> it quite frequently. I really can't take Philip's approach (I don't own
> the repository). :-(
Since this the solution to this is non-obvious to me at this point I
hope to find some time at the Bazaar Rodeo early January to discuss and
fix it.

Jesse (sbjesse) wrote on 2011-01-09:

#29

i am bit by this too. i've been tracking chromium source repo for a few months and only have lagged behind recently. a pull from svn will need more memory than i have so i decided to pull from lp:~vcs-imports/chromium-browser/trunk then the glorious crash came ... so sad

Jelmer Vernooij (jelmer) on 2011-03-10

Changed in bzr:
status:	Incomplete → Invalid
Changed in bzr-svn:
status:	Incomplete → Triaged
importance:	Undecided → High

Jelmer Vernooij (jelmer) on 2011-03-30

Changed in bzr-svn:
milestone:	none → 1.1.0
assignee:	nobody → Jelmer Vernooij (jelmer)

Jelmer Vernooij (jelmer) on 2011-04-04

Changed in bzr-svn:
status:	Triaged → In Progress

Jelmer Vernooij (jelmer) on 2011-04-24

Changed in bzr-svn:
importance:	High → Critical

Gordon Tyler (doxxx) wrote on 2011-05-13:

#30

Just got hit by this. I have a shared repo (colo workspace) into which I have branched a svn trunk. I worked with this for months, creating local feature branches and merging them back into trunk and vice versa, and pushing up to svn trunk.

Then yesterday I branched the svn trunk on the svn server directly using TortoiseSVN and then branched that into my local bzr shared repo. I cherry-picked a few changes from the bzr trunk into the bzr branch, committed and pushed up to teh svn branch without trouble. Today, I tried to do the same thing again, cherry-pick a revision from trunk into the branch and it gave me the BzrCheckError.

I'm running bzr 2.3.0 and bzr-svn 1.0.5dev (as bundled with bzr 2.3.0).

Adrian Wilkins (adrian-wilkins) wrote on 2011-07-13:

#31

Was also struck by this.

Where M1 is my desktop and M2 is my laptop

M1 :
merged some changes from bzr topic branch into SVN trunk
pushed these to SVN

# somewhat later
M2 :
pulled this merged revision from SVN
also pulled some later revisions committed to SVN by other SVN clients

M1 :
attempt to pull branch from M2
error occurs

M1 :
  in attempt to fix missing ghosts, tried push --overwrite of this branch to M2 (over bzr:// smart server)
  this does not fix the problem ; presumably M2 just reports that it already has the tip revision and nothing actually happens
  would this work if you tried over file:// ?

Resolution :
M2 :
  delete repository
  create new empty repository
M1 :
  push SVN trunk branch to M2 repository
M2 :
  pull new SVN revisions
  # only M2 has access to SVN at the moment because the VPN concentrator is prejudiced against non-Windows VPN clients
M1 :
  pull new SVN revisions from bzr branch on M2
  success!

I can imagine that where multiple persons are merging and then pushing to SVN, this would be a real showstopper.

You should be able to prevent this by only pushing Bazaar revisions to SVN from one machine, or by using rebase exclusively. This has not happened to my colleagues who only use rebase. It might be simplest to set up a bzr repository on the SVN server that pulls desired SVN branches via SVN post_commit hooks and pushes back any revisions it receives to these branches.

Would a command to deliberately lock a bzr branch assist this use case? Some SVN commits can be quite prolonged and it would be annoying to push a revision, only to discover it had then been overwritten by a pull from the SVN branch (--overwrite being pretty much a given, since you cannot rewrite SVN history). The lock command could be issued by the SVN pre-commit hook, then released before the post-commit pull. Although a quick and dirty echo of a lock file might work.

Bazaar (bzr) 2.3.1 bzr-svn 1.0.5dev # standard Natty packages

Jelmer Vernooij (jelmer) on 2011-08-21

Changed in bzr-svn:
status:	In Progress → Fix Committed

Jelmer Vernooij (jelmer) wrote on 2011-08-21:

#32

This should now *finally* be fixed. bzr-svn will now always store the right text revisions and text parents, and has a test framework to test the various scenarios.

This means that bzr-svn should now always write correct data to svn repositories. There might still be a few out there with incorrect metadata that make bzr-svn blow up; bug 628354 is about that.

Jelmer Vernooij (jelmer) on 2011-08-26

Changed in bzr-svn:
status:	Fix Committed → Fix Released

Patrick Noffke (patrick-noffke) wrote on 2011-09-13:

#33

Download full text (4.1 KiB)

I just got this error when attempting to merge from another local branch:

$ bzr merge ../trunk-bzr-svn
bzr: ERROR: bzrlib.errors.BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps

Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/bzrlib/commands.py", line 927, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/bzrlib/commands.py", line 1127, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib64/python2.7/site-packages/bzrlib/commands.py", line 692, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib64/python2.7/site-packages/bzrlib/commands.py", line 714, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/bzrlib/builtins.py", line 3941, in run
    location, revision, remember, possible_transports, None)
  File "/usr/lib64/python2.7/site-packages/bzrlib/builtins.py", line 4074, in _get_merger_from_branch
    other_revision_id, base_revision_id, other_branch, base_branch)
  File "/usr/lib64/python2.7/site-packages/bzrlib/merge.py", line 410, in from_revision_ids
    merger.set_other_revision(other, other_branch)
  File "/usr/lib64/python2.7/site-packages/bzrlib/merge.py", line 551, in set_other_revision
    self._maybe_fetch(other_branch, self.this_branch, self.other_rev_id)
  File "/usr/lib64/python2.7/site-packages/bzrlib/merge.py", line 568, in _maybe_fetch
    target.fetch(source, revision_id)
  File "/usr/lib64/python2.7/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/bzrlib/branch.py", line 687, in fetch
    pb=pb)
  File "/usr/lib64/python2.7/site-packages/bzrlib/repository.py", line 1781, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/usr/lib64/python2.7/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/bzrlib/repository.py", line 3437, in fetch
    find_ghosts=find_ghosts)
  File "/usr/lib64/python2.7/site-packages/bzrlib/fetch.py", line 73, in __init__
    self.__fetch()
  File "/usr/lib64/python2.7/site-packages/bzrlib/fetch.py", line 99, in __fetch
    self._fetch_everything_for_search(search)
  File "/usr/lib64/python2.7/site-packages/bzrlib/fetch.py", line 127, in _fetch_everything_for_search
    stream, from_format, [])
  File "/usr/lib64/python2.7/site-packages/bzrlib/repository.py", line 4105, in insert_stream
    hint = self.target_repo.commit_write_group()
  File "/usr/lib64/python2.7/site-packages/bzrlib/repository.py", line 1640, in commit_write_group
    result = self._commit_write_group()
  File "/usr/lib64/python2.7/site-...

Bazaar

missing chk node(s) for id_to_entry maps

Bug Description

Related branches

Duplicates of this bug

Other bug subscribers

Bug attachments