Bazaar

ObjectNotLocked: _KnitGraphIndex(CombinedGraphIndex()) is not locked running diff across hpss

Bug #389413 reported by Martin Pool
32
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Bazaar
Fix Released
High
Andrew Bennetts
Bazaar 2.1.0b1 "While the cat is away"
2.0
Fix Released
High
Andrew Bennetts
Bazaar 2.0.1 "Stability First"

Bug Description

mbp@grace [1 job]% ./bzr diff -c 4148 lp:~mbp/bzr/progress
Unable to load plugin 'gtk'. It requested API version (1, 15, 0) of module <module 'bzrlib' from '/home/mbp/bzr/progress/bzrlib/__init__.pyc'> but the minimum exported version is (1, 17, 0), and the maximum is (1, 17, 0)
bzr: ERROR: bzrlib.errors.ObjectNotLocked: _KnitGraphIndex(CombinedGraphIndex()) is not locked

Traceback (most recent call last):
  File "/home/mbp/bzr/progress/bzrlib/commands.py", line 831, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/home/mbp/bzr/progress/bzrlib/commands.py", line 1026, in run_bzr
    ret = run(*run_argv)
  File "/home/mbp/bzr/progress/bzrlib/commands.py", line 643, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/mbp/bzr/progress/bzrlib/commands.py", line 1041, in ignore_pipe
    result = func(*args, **kwargs)
  File "/home/mbp/bzr/progress/bzrlib/builtins.py", line 1836, in run
    apply_view=True)
  File "/home/mbp/bzr/progress/bzrlib/diff.py", line 338, in _get_trees_to_diff
    old_tree = _get_tree_to_diff(old_revision_spec, working_tree, branch)
  File "/home/mbp/bzr/progress/bzrlib/diff.py", line 389, in _get_tree_to_diff
    return spec.as_tree(branch)
  File "/home/mbp/bzr/progress/bzrlib/revisionspec.py", line 260, in as_tree
    return self._as_tree(context_branch)
  File "/home/mbp/bzr/progress/bzrlib/revisionspec.py", line 269, in _as_tree
    revision_id = self.as_revision_id(context_branch)
  File "/home/mbp/bzr/progress/bzrlib/revisionspec.py", line 243, in as_revision_id
    return self._as_revision_id(context_branch)
  File "/home/mbp/bzr/progress/bzrlib/revisionspec.py", line 534, in _as_revision_id
    base_revision_id = base_revspec.as_revision_id(context_branch)
  File "/home/mbp/bzr/progress/bzrlib/revisionspec.py", line 243, in as_revision_id
    return self._as_revision_id(context_branch)
  File "/home/mbp/bzr/progress/bzrlib/revisionspec.py", line 387, in _as_revision_id
    branch, revno, revision_id = self._lookup(context_branch, None)
  File "/home/mbp/bzr/progress/bzrlib/revisionspec.py", line 380, in _lookup
    revision_id = branch.get_rev_id(revno, revs_or_none)
  File "/home/mbp/bzr/progress/bzrlib/remote.py", line 2266, in get_rev_id
    revno, last_revision_info)
  File "/home/mbp/bzr/progress/bzrlib/remote.py", line 691, in get_rev_id_for_revno
    return self._get_rev_id_for_revno_vfs(revno, known_pair)
  File "/home/mbp/bzr/progress/bzrlib/remote.py", line 679, in _get_rev_id_for_revno_vfs
    revno, known_pair)
  File "/home/mbp/bzr/progress/bzrlib/repository.py", line 2277, in get_rev_id_for_revno
    self, partial_history, stop_index=distance_from_known)
  File "/home/mbp/bzr/progress/bzrlib/repository.py", line 4358, in _iter_for_revno
    iterator.next()
  File "/home/mbp/bzr/progress/bzrlib/repository.py", line 2307, in iter_reverse_revision_history
    parents = graph.get_parent_map([next_id])[next_id]
  File "/home/mbp/bzr/progress/bzrlib/graph.py", line 167, in get_parent_map
    parent_map = self._get_parent_map(needed_revisions)
  File "/home/mbp/bzr/progress/bzrlib/repository.py", line 2458, in get_parent_map
    self.revisions.get_parent_map(query_keys).iteritems():
  File "/home/mbp/bzr/progress/bzrlib/knit.py", line 1153, in get_parent_map
    return self._get_parent_map_with_sources(keys)[0]
  File "/home/mbp/bzr/progress/bzrlib/knit.py", line 1172, in _get_parent_map_with_sources
    new_result = source.get_parent_map(missing)
  File "/home/mbp/bzr/progress/bzrlib/knit.py", line 3012, in get_parent_map
    self._check_read()
  File "/home/mbp/bzr/progress/bzrlib/knit.py", line 2895, in _check_read
    raise errors.ObjectNotLocked(self)
ObjectNotLocked: _KnitGraphIndex(CombinedGraphIndex()) is not locked

bzr 1.17dev on python 2.6.2 (linux2)
arguments: ['./bzr', 'diff', '-c', '4148', 'lp:~mbp/bzr/progress']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_AU.UTF-8'
plugins:
  explorer /home/mbp/.bazaar/plugins/explorer [0.1dev]
  launchpad /home/mbp/bzr/progress/bzrlib/plugins/launchpad [1.17dev]
  netrc_credential_store /home/mbp/bzr/progress/bzrlib/plugins/netrc_credential_store [1.17dev]
  pqm /home/mbp/.bazaar/plugins/pqm [1.3dev]
  qbzr /home/mbp/.bazaar/plugins/qbzr [0.11dev]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

Tags: hpss

Related branches

lp:~spiv/bzr/hpss-diff-locking-bug
Vincent Ladeuil: Approve
Diff: 55 lines
3 files modified
NEWS (+3/-0)
bzrlib/remote.py (+1/-0)
bzrlib/tests/test_remote.py (+20/-0)
lp:ubuntu/karmic-proposed/bzr
Revision history for this message
Martin Pool (mbp) wrote :
Download full text (13.6 KiB)

Here's the hpss trace in case that helps.

It's reproducible, at least on this branch.

Fri 2009-06-19 19:51:01 +1000
0.026 bzr arguments: [u'-Dhpss', u'diff', u'-c', u'4148', u'lp:~mbp/bzr/progress']
0.043 looking for plugins in /home/mbp/.bazaar/plugins
[16744] 2009-06-19 19:51:02.001 WARNING: Unable to load plugin 'gtk'. It requested API version (1, 15, 0) of module <module 'bzrlib' from '/home/mbp/bzr/progress/bzrlib/__init__.pyc'> but the minimum exported version is (1, 17, 0), and the maximum is (1, 17, 0)
0.121 looking for plugins in /home/mbp/bzr/progress/bzrlib/plugins
0.142 looking for plugins in /usr/lib/python2.6/dist-packages/bzrlib/plugins
0.143 Plugin name netrc_credential_store already loaded
0.143 Plugin name launchpad already loaded
0.148 Returning RevisionSpec RevisionSpec_before for before:4148
2.402 hpss: Built a new medium: SmartSSHClientMedium
2.427 hpss call: 'BzrDir.open', '~mbp/bzr/progress/'
2.427 (to bzr+ssh://bazaar.launchpad.net/%7Embp/bzr/progress/)
2.474 ssh implementation is OpenSSH
10.193 result: ('yes',)
10.194 hpss call: 'BzrDir.open_branchV2', '~mbp/bzr/progress/'
10.194 (to bzr+ssh://bazaar.launchpad.net/%7Embp/bzr/progress/)
10.564 result: ('branch', 'Bazaar Branch Format 7 (needs bzr 1.6)\n')
10.564 hpss call: 'BzrDir.find_repositoryV3', '~mbp/bzr/progress/'
10.564 (to bzr+ssh://bazaar.launchpad.net/%7Embp/bzr/progress/)
10.931 result: ('ok', '', 'no', 'no', 'yes', 'Bazaar RepositoryFormatKnitPack6 (bzr 1.9)\n')
10.954 hpss call: 'Branch.get_stacked_on_url', '~mbp/bzr/progress/'
10.955 (to bzr+ssh://bazaar.launchpad.net/%7Embp/bzr/progress/)
11.339 result: ('ok', '/~bzr/bzr/trunk')
11.340 hpss call: 'BzrDir.open', '~bzr/bzr/trunk/'
11.340 (to bzr+ssh://bazaar.launchpad.net/%7Embp/bzr/progress/)
11.799 result: ('yes',)
11.799 hpss call: 'BzrDir.open_branchV2', '~bzr/bzr/trunk/'
11.799 (to bzr+ssh://bazaar.launchpad.net/%7Embp/bzr/progress/)
12.168 result: ('branch', 'Bazaar Branch Format 7 (needs bzr 1.6)\n')
12.168 hpss call: 'BzrDir.find_repositoryV3', '~bzr/bzr/trunk/'
12.168 (to bzr+ssh://bazaar.launchpad.net/%7Embp/bzr/progress/)
12.534 result: ('ok', '', 'no', 'no', 'yes', 'Bazaar RepositoryFormatKnitPack6 (bzr 1.9)\n')
12.535 hpss call: 'Branch.get_stacked_on_url', '~bzr/bzr/trunk/'
12.535 (to bzr+ssh://bazaar.launchpad.net/%7Embp/bzr/progress/)
12.906 result: ('NotStacked',)
12.907 hpss call: 'Branch.last_revision_info', '~mbp/bzr/progress/'
12.907 (to bzr+ssh://bazaar.launchpad.net/%7Embp/bzr/progress/)
13.281 result: ('ok', '4148', '<email address hidden>')
13.282 hpss call: 'Branch.last_revision_info', '~mbp/bzr/progress/'
13.282 (to bzr+ssh://bazaar.launchpad.net/%7Embp/bzr/progress/)
13.654 result: ('ok', '4148', '<email address hidden>')
13.655 hpss call: 'Repository.get_rev_id_for_revno', '~mbp/bzr/progress/', 4148, (4148, '<email address hidden>')
13.655 ...

Revision history for this message
Martin Pool (mbp) wrote :

It also occurs when run with arguments: ['./bzr', '-Dhpss', 'diff', '-c', '4148', 'lp:bzr'] on lp:bzr which should not be stacked therefore might avoid some problems.

This command _does_ work with bzr nightly 1.15+4441+115 but it fails using my copy of trunk on revno 4463. So it seems like a recent regression.

Revision history for this message
Andrew Bennetts (spiv) wrote :

The current instance of this is probably fallout from my recent get_rev_id_for_revno patch (and so is probably actually a seperate bug to this original report, but nevermind...). I'll look into this on Monday, it's probably fairly shallow.

Changed in bzr:
assignee: nobody → Andrew Bennetts (spiv)
importance: Medium → High
Revision history for this message
Jean-Francois Roy (jeanfrancois.roy) wrote :

This also occurred to me with a log -r -1 command on an LP-hosted branch.

Revision history for this message
Benoit Pierre (benoit.pierre) wrote :
Download full text (3.9 KiB)

I get a similar error on a local 2a branch when running ' bzr log path/to/file -r 199'

bzr: ERROR: bzrlib.errors.ObjectNotLocked: <bzrlib.groupcompress._GCGraphIndex object at 0x3221d90> is not locked

Traceback (most recent call last):
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/commands.py", line 835, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/commands.py", line 1030, in run_bzr
    ret = run(*run_argv)
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/commands.py", line 647, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/commands.py", line 1045, in ignore_pipe
    result = func(*args, **kwargs)
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/builtins.py", line 2262, in run
    file_list)
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/log.py", line 1897, in _get_info_for_log_files
    kind = _get_kind_for_file_id(tree, file_id)
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/log.py", line 1929, in _get_kind_for_file_id
    return tree.kind(file_id)
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/revisiontree.py", line 148, in kind
    return self._inventory[file_id].kind
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/inventory.py", line 1875, in __getitem__
    self.id_to_entry.iteritems([(file_id,)]).next()[1])
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/chk_map.py", line 484, in iteritems
    self._ensure_root()
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/chk_map.py", line 138, in _ensure_root
    self._root_node = self._get_node(self._root_node)
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/chk_map.py", line 151, in _get_node
    bytes = self._read_bytes(node)
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/chk_map.py", line 162, in _read_bytes
    bytes = stream.next().get_bytes_as('fulltext')
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/groupcompress.py", line 1193, in get_record_stream
    orig_keys, ordering, include_delta_closure):
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/groupcompress.py", line 1298, in _get_remaining_record_stream
    locations = self._index.get_build_details(keys)
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/groupcompress.py", line 1767, in get_build_details
    self._check_read()
  File "/home/bpierre/progs/lib/python2.5/site-packages/bzrlib/groupcompress.py", line 1691, in _check_read
    raise errors.ObjectNotLocked(self)
ObjectNotLocked: <bzrlib.groupcompress._GCGraphIndex object at 0x3221d90> is not locked

bzr 2.0dev on python 2.5.4 (linux2)
arguments: ['/home/bpierre/progs/bin/bzr', 'log', 'path/to/file', '-r', '199']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bzrtools /home/bpierre/.bazaar/plugins/bzrtools [2.0]
  diffstat /home/bpierre/.bazaar/plugins/diffstat [0.2]
  explorer /home/bpierre/.bazaar/plugins/explorer [0.7dev]
  extmerge /home/bp...

Read more...

Revision history for this message
Benoit Pierre (benoit.pierre) wrote :
Download full text (3.7 KiB)

And when doing a checkout of a 1.6 branch:

/usr/bin/bzr checkout --revision 851 bzr+ssh://server/trunk build

bzr: ERROR: bzrlib.errors.ObjectNotLocked: _KnitGraphIndex(CombinedGraphIndex()) is not locked

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 835, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 1030, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 647, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 1316, in run
    revision_id = revision.as_revision_id(source)
  File "/usr/lib/python2.5/site-packages/bzrlib/revisionspec.py", line 243, in as_revision_id
    return self._as_revision_id(context_branch)
  File "/usr/lib/python2.5/site-packages/bzrlib/revisionspec.py", line 387, in _as_revision_id
    branch, revno, revision_id = self._lookup(context_branch, None)
  File "/usr/lib/python2.5/site-packages/bzrlib/revisionspec.py", line 380, in _lookup
    revision_id = branch.get_rev_id(revno, revs_or_none)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 2271, in get_rev_id
    revno, last_revision_info)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 696, in get_rev_id_for_revno
    return self._get_rev_id_for_revno_vfs(revno, known_pair)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 684, in _get_rev_id_for_revno_vfs
    revno, known_pair)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2269, in get_rev_id_for_revno
    self, partial_history, stop_index=distance_from_known)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 4370, in _iter_for_revno
    iterator.next()
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2299, in iter_reverse_revision_history
    parents = graph.get_parent_map([next_id])[next_id]
  File "/usr/lib/python2.5/site-packages/bzrlib/graph.py", line 167, in get_parent_map
    parent_map = self._get_parent_map(needed_revisions)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2457, in get_parent_map
    self.revisions.get_parent_map(query_keys).iteritems():
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1193, in get_parent_map
    return self._get_parent_map_with_sources(keys)[0]
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1212, in _get_parent_map_with_sources
    new_result = source.get_parent_map(missing)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 3047, in get_parent_map
    self._check_read()
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 2930, in _check_read
    raise errors.ObjectNotLocked(self)
ObjectNotLocked: _KnitGraphIndex(CombinedGraphIndex()) is not locked

bzr 1.17 on python 2.5.2 (linux2)
arguments: ['/usr/bin/bzr', 'checkout', '--revision', '851', 'bzr+ssh://server/trunk', 'build']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.17]
  la...

Read more...

Revision history for this message
Pieter Nagel (pieter-equinox) wrote :
Download full text (6.5 KiB)

This also happens to us in multiple scenarios:

------------
When we do a reverse merge to revert a specific change:

bzr merge -r4945..4944
Merging from remembered submit location bzr+ssh://pixar/devel/bzr/dante-head/
Merging from remembered submit location bzr+ssh://pixar/devel/bzr/dante-head/
bzr: ERROR: bzrlib.errors.ObjectNotLocked: _KnitGraphIndex(CombinedGraphIndex()) is not locked

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 835, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 1030, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 647, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/qbzr/lib/commands.py", line 671, in run
    return bzrlib.builtins.cmd_merge.run(self, *args, **kw)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 3681, in run
    location, revision, remember, possible_transports, pb)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 3795, in _get_merger_from_branch
    other_revision_id = revision[-1].as_revision_id(other_branch)
  File "/usr/lib/python2.5/site-packages/bzrlib/revisionspec.py", line 243, in as_revision_id
    return self._as_revision_id(context_branch)
  File "/usr/lib/python2.5/site-packages/bzrlib/revisionspec.py", line 387, in _as_revision_id
    branch, revno, revision_id = self._lookup(context_branch, None)
  File "/usr/lib/python2.5/site-packages/bzrlib/revisionspec.py", line 380, in _lookup
    revision_id = branch.get_rev_id(revno, revs_or_none)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 2317, in get_rev_id
    revno, last_revision_info)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 700, in get_rev_id_for_revno
    return self._get_rev_id_for_revno_vfs(revno, known_pair)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 688, in _get_rev_id_for_revno_vfs
    revno, known_pair)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2427, in get_rev_id_for_revno
    self, partial_history, stop_index=distance_from_known)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 4549, in _iter_for_revno
    iterator.next()
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2457, in iter_reverse_revision_history
    parents = graph.get_parent_map([next_id])[next_id]
  File "/usr/lib/python2.5/site-packages/bzrlib/graph.py", line 167, in get_parent_map
    parent_map = self._get_parent_map(needed_revisions)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2615, in get_parent_map
    self.revisions.get_parent_map(query_keys).iteritems():
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1200, in get_parent_map
    return self._get_parent_map_with_sources(keys)[0]
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1219, in _get_parent_map_with_sources
    new_result = source.get_parent_map(missing)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 3059, in...

Read more...

Revision history for this message
Andrew Bennetts (spiv) wrote :

It's a bit harder to reproduce this now that Launchpad has upgraded its bzr. But it's still easy to provoke with a smart server that doesn't support the "Repository.get_rev_id_for_revno" verb (or by hacking remote.py locally to make it seem that the server doesn't support it).

The root cause I think is that the branch/repo really ought to be read locked well before this point, by cmd_diff's main function or at least revisionspec.py. I see that revisionspec.py actually does its own Branch.open call in some circumstances though...

The bandaid fix would be to put @needs_read_lock on some more methods in remote.py. But I'll see how hard it is to fix the root problem first.

I'm not sure if the other reports in the comments about "bzr log" etc are the same bug or not yet.

Changed in bzr:
status: Confirmed → In Progress
Revision history for this message
Andrew Bennetts (spiv) wrote :

I've decided that a good immediate fix is to apply @need_read_lock to RemoteBranch.get_rev_id, as this is consistent with Branch.get_rev_id.

There's a broader issue that we really shouldn't be repeatedly locking and unlocking these objects just to do diff and log, but that's a separate problem to making it work.

Also, Benoit PIERRE @5: I think you have a different bug to everyone else on this bug report, as yours doesn't involve the smart server. I can reproduce it, so I'll file a separate bug about it.

Changed in bzr:
status: In Progress → Fix Committed
Revision history for this message
Andrew Bennetts (spiv) wrote :

I've filed <https://bugs.launchpad.net/bzr/+bug/445171> for the bug reported in comment 5.

Revision history for this message
Andrew Bennetts (spiv) wrote :

Fix landed on 2.0 branch. It should be part of 2.0.1.

John A Meinel (jameinel)
Changed in bzr:
milestone: none → 2.1.0b1
Andrew Bennetts (spiv)
Changed in bzr:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.