KeyError in object_store.py

[Brian de Alwis]

I was trying to update a previously-pulled branch and bzr git threw a KeyError (see below). This branch was previously part of a repository that was made standalone, then moved, and then colo-ify'd (with bzr-colo).

What's interesting is that I was able to successfully re-branch from the original repository (on github) to a new local branch, and subsequently pull from that new local branch. And a subsequent re-pull from the original parent on github worked fine.

$ bzr pull
Using saved parent location: git://github.com/sonatype/sonatype-tycho.git
bzr: ERROR: exceptions.KeyError: '7e89500b6047cc00a240919ff181cae6dbc9b69b'

Traceback (most recent call last):
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/commands.py", line 907, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/commands.py", line 1111, in run_bzr
    ret = run(*run_argv)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/commands.py", line 685, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/commands.py", line 700, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/cleanup.py", line 122, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/cleanup.py", line 156, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/builtins.py", line 1050, in run
    possible_transports=possible_transports, local=local)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/workingtree.py", line 1608, in pull
    local=local)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/branch.py", line 1009, in pull
    possible_transports=possible_transports, *args, **kwargs)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/git/branch.py", line 510, in pull
    stop_revision, overwrite=overwrite, graph=graph, limit=limit)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/git/branch.py", line 459, in _update_revisions
    determine_wants, self.source.mapping, limit=limit)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/git/fetch.py", line 540, in fetch_objects
    objects_iter, store, recorded_wants, pb, limit)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/git/fetch.py", line 431, in import_git_objects
    target_git_object_retriever, trees_cache)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/git/fetch.py", line 348, in import_git_commit
    allow_submodules=getattr(repo._format, "supports_tree_reference", False))
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/git/fetch.py", line 259, in import_git_tree
    allow_submodules=allow_submodules)
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/git/fetch.py", line 281, in import_git_tree
    base_tree, existing_children, lookup_object))
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/git/fetch.py", line 204, in remove_disappeared_children
    base_inv, c_path, lookup_object(hexsha), [], lookup_object))
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/git/fetch.py", line 204, in remove_disappeared_children
    base_inv, c_path, lookup_object(hexsha), [], lookup_object))
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/git/fetch.py", line 384, in lookup_object
    return target_git_object_retriever[sha]
  File "/usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/git/object_store.py", line 560, in __getitem__
    raise KeyError(sha)
KeyError: '7e89500b6047cc00a240919ff181cae6dbc9b69b'

bzr 2.2b2 on python 2.6.2 (Darwin-10.4.0-i386-32bit)
arguments: ['/usr/pkg/bin/bzr', 'pull']
encoding: 'UTF-8', fsenc: 'utf-8', lang: 'en_CA.UTF-8'
plugins:
  automv /Users/bsd/.bazaar/plugins/automv [unknown]
  bash_completion /Users/bsd/.bazaar/plugins/bash_completion [1.1.0dev]
  bzrtools /Users/bsd/.bazaar/plugins/bzrtools [2.1b1]
  clear_exec_bit /Users/bsd/.bazaar/plugins/clear_exec_bit.py [unknown]
  colo /Users/bsd/.bazaar/plugins/colo [0.2.0dev]
  diffstat /Users/bsd/.bazaar/plugins/diffstat [0.2.0]
  externals /Users/bsd/.bazaar/plugins/externals [1.1.0]
  extmerge /Users/bsd/.bazaar/plugins/extmerge [unknown]
  git /usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/git [0.5.1]
  grep /Users/bsd/.bazaar/plugins/grep [0.4.0]
  launchpad /usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/launchpad [2.2b2]
  mail_app /Users/bsd/.bazaar/plugins/mail_app [0.9.0]
  netrc_credential_store /usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/netrc_credential_store [2.2b2]
  news_merge /usr/pkg/lib/python2.6/site-packages/bzrlib/plugins/news_merge [2.2b2]
  rewrite /Users/bsd/.bazaar/plugins/rewrite [0.6.1dev]
  sprout /Users/bsd/.bazaar/plugins/sprout [0.9.0dev]
  svn /Users/bsd/.bazaar/plugins/svn [1.0.3dev]
  xmloutput /Users/bsd/.bazaar/plugins/xmloutput [0.8.6.dev]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.

[Jelmer Vernooij (jelmer)]

Attempting this with a trunk version of bzr-git might fix this issue,
though I doubt that will be the case.

Does "bzr git-objects" work without problems ?

Another thing you could try is to remove the bzr-git cache
at .bzr/repository/git and trying again. That'll cause bzr-git to
rebuild the cache.

Cheers,

Jelmer

[Brian de Alwis (slyguy)]

I remembered that I still had the original repository. Experimenting some more, I've found how to reproduce the problem:

  1. bzr branch git://github.com/sonatype/sonatype-tycho.git tycho-trunk
  2. bzr branch -r git:e3496cbba2628c05a8d03d6a95af026dab1d3fa2 tycho-trunk tycho-past
  3. cd tycho-past
  4. bzr colo-ify
       Note that 'bzr git-objects' runs fine (when run from .bzr.branches, the repository).
  5. bzr pull git://github.com/sonatype/sonatype-tycho.git

And the problem recurs if I "rm -rf .bzr/branches/.bzr/repository/git" and then repull.

[Jelmer Vernooij (jelmer)]

On Thu, 2010-06-24 at 18:37 +0000, Brian de Alwis wrote:
> I remembered that I still had the original repository. Experimenting
> some more, I've found how to reproduce the problem:
>
> 1. bzr branch git://github.com/sonatype/sonatype-tycho.git tycho-trunk
> 2. bzr branch -r git:e3496cbba2628c05a8d03d6a95af026dab1d3fa2 tycho-trunk tycho-past
> 3. cd tycho-past
> 4. bzr colo-ify
> Note that 'bzr git-objects' runs fine (when run from .bzr.branches, the repository).
> 5. bzr pull git://github.com/sonatype/sonatype-tycho.git
>
> And the problem recurs if I "rm -rf .bzr/branches/.bzr/repository/git"
> and then repull.
Can you reproduce this without involving colo-ify ? If not then I expect
this to be an issue in bzr-colo somewhere.

colo-ify somehow seems to create a cache database for bzr-git or perhaps
creates ghosts.

The code path that causes this exception to be raised is only executed
based on information in the cache, and that cache should not exist yet
(since you have a vanilla branch, created by "bzr branch").

Cheers,

Jelmer

[Brian de Alwis (slyguy)]

It does seem to be specific to bzr-colo, unfortunately. I've added Neil Martinsen-Burrell to the bug, in case he has any insights.

[Neil Martinsen-Burrell (nmb)]

I won't have time for a week or so to look into this, but the steps used by colo-ify should be reproducible using simple bzr commands and moving directories. Try replacing step 3 by:

3a. mkdir .bzr/branches
3b. bzr branch . .bzr/branches/trunk
3c. bzr reconfigure --lightweight-checkout --bind-to=.bzr/branches/trunk

and see what happens. If that doesn't reproduce the bug, then there's a bug in colo-ify.

[Alexander Belchenko (bialix)]

Neil Martinsen-Burrell пишет:
> I won't have time for a week or so to look into this, but the steps used
> by colo-ify should be reproducible using simple bzr commands and moving
> directories. Try replacing step 3 by:
>
> 3a. mkdir .bzr/branches

I think this step actually should be

bzr init-repo .bzr/branches --no-trees

> 3b. bzr branch . .bzr/branches/trunk
> 3c. bzr reconfigure --lightweight-checkout --bind-to=.bzr/branches/trunk
>
> and see what happens. If that doesn't reproduce the bug, then there's a
> bug in colo-ify.

IIUC, you're using bzrlib API for this. So the problem in API usage.
I'm really doubt there is any bugs in colo-ify.

[Neil Martinsen-Burrell (nmb)]

It appears to work with substituting 3a, 3b and 3c for colo-ify. The thing that happened on the pull was "Updating git map" for all 800 revisions of that branch.

It appears that there is an inconsistency between what we do in colo-ify using the API and what init-repo;branch;reconfigure does.

I'll next be able to work on this in about a week.

[Jelmer Vernooij (jelmer)]

If colo purges revisions from the repository during colo-ify, then this is most likely what is causing the issue.

[Neil Martinsen-Burrell (nmb)]

colo-ify shouldn't be purging revisions, although it does make a new repository and branch the existing branch into that repository using BzrDir.sprout. The relevant code in colo-ify is:

def run(self, other_location_list=[], trunk_name='trunk'):
        the_bzrdir, relpath = bzrdir.BzrDir.open_containing(u'.')
        the_bzrdir.root_transport.mkdir(COLOCATED_LOCATION)
        repo_location = urlutils.join(the_bzrdir.root_transport.base,
                          COLOCATED_LOCATION)
        new_transport = transport.get_transport(repo_location,
            possible_transports=[the_bzrdir.root_transport])
        new_bzrdir = the_bzrdir.cloning_metadir().initialize_on_transport(
                         new_transport)
        new_repo = new_bzrdir.create_repository(shared=True)
        new_repo.set_make_working_trees(False)

        the_branch = _mod_branch.Branch.open_containing(u'.')[0]
        branch_location = urlutils.join(repo_location, trunk_name)
        new_branch_bzrdir = the_bzrdir.sprout(branch_location,
                                       source_branch=the_branch,
                                       create_tree_if_local=False)
        new_branch = new_branch_bzrdir.open_branch()
        new_branch.set_parent(None)
        self._copy_locations(the_branch, new_branch)

        reconfiguration = reconfigure.Reconfigure.to_lightweight_checkout(
                              the_bzrdir, reference_location=branch_location)
        reconfiguration.apply()

Am I misusing one of the API calls here? sprout()? It looks like BzrDir.sprout eventually runs Repository.fetch to get the revisions into the newly created repository. Could that not be pulling in all of the necessary revisions? Seems very unlikely...