Unable to clone percona-xtrabackup

[Nirbhay Choubey]

bzr crashes with "missing referenced chk root keys:" while cloning the tree.
I have tried it multiple times with same result.

$ cat /etc/issue
Ubuntu 12.04.4 LTS \n \l

$ uname -a
Linux nirbhay-VirtualBox 3.8.0-42-generic #62~precise1-Ubuntu SMP Wed Jun 4 22:06:29 UTC 2014 i686 i686 i386 GNU/Linux

$ bzr --version
Bazaar (bzr) 2.5.1
  Python interpreter: /usr/bin/python 2.7.3
  Python standard library: /usr/lib/python2.7
  Platform: Linux-3.8.0-42-generic-i686-with-Ubuntu-12.04-precise
...

nirbhay@nirbhay-VirtualBox:~/project/repo/percona$ bzr branch lp:percona-xtrabackup
bzr: ERROR: bzrlib.errors.BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing referenced chk root keys: [StaticTuple('sha1:3cbc15851d82ad932e73ba8b79db765e0e280b75',)]

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 920, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 1131, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 673, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 695, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 136, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 166, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/builtins.py", line 1493, in run
    source_branch=br_from)
  File "/usr/lib/python2.7/dist-packages/bzrlib/bzrdir.py", line 366, in sprout
    create_tree_if_local=create_tree_if_local)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 132, in run
    self.cleanups, self.func, self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 166, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/bzrdir.py", line 416, in _sprout
    result_repo.fetch(source_repository, fetch_spec=fetch_spec)
  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 1267, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/usr/lib/python2.7/dist-packages/bzrlib/decorators.py", line 218, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 2584, in fetch
    find_ghosts=find_ghosts)
  File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 77, in __init__
    self.__fetch()
  File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 104, in __fetch
    self._fetch_everything_for_search(search_result)
  File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 132, in _fetch_everything_for_search
    stream, from_format, [])
  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 2050, in insert_stream
    hint = self.target_repo.commit_write_group()
  File "/usr/lib/python2.7/dist-packages/bzrlib/repository.py", line 633, in commit_write_group
    result = self._commit_write_group()
  File "/usr/lib/python2.7/dist-packages/bzrlib/repofmt/pack_repo.py", line 1727, in _commit_write_group
    hint = self._pack_collection._commit_write_group()
  File "/usr/lib/python2.7/dist-packages/bzrlib/repofmt/pack_repo.py", line 1607, in _commit_write_group
    "Cannot add revision(s) to repository: " + problems_summary)
BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing referenced chk root keys: [StaticTuple('sha1:3cbc15851d82ad932e73ba8b79db765e0e280b75',)]

[George Ormond Lorch III (gl-az)]

I believe this is a bzr issues that we all ran into a while back. You should to upgrade to bzr 2.06. The bzr bug can be found here: https://bugs.launchpad.net/bzr/+bug/522637 and is mentioned in the 2.06 release notes here: http://doc.bazaar.canonical.com/bzr.2.0/en/release-notes/bzr-2.0.6.html

[Nirbhay Choubey (nirbhay)]

Upgrade to 2.0.6? But I already using 2.5.1.

[Valerii Kravchuk (valerii-kravchuk)]

Not that I am on Ubuntu 12.04 and exactly the same bzr version, but I can not reproduce the problem:

[openxs@chief bzr]$ bzr --version
Bazaar (bzr) 2.2.4
  Python interpreter: /usr/bin/python 2.7.0
  Python standard library: /usr/lib64/python2.7
  Platform: Linux-2.6.35.14-106.fc14.x86_64-x86_64-with-fedora-14-Laughlin
  bzrlib: /usr/lib64/python2.7/site-packages/bzrlib
  Bazaar configuration: /home/openxs/.bazaar
  Bazaar log file: /home/openxs/.bzr.log

Copyright 2005-2010 Canonical Ltd.
http://bazaar.canonical.com/

bzr comes with ABSOLUTELY NO WARRANTY. bzr is free software, and
you may use, modify and redistribute it under the terms of the GNU
General Public License version 2 or later.

Bazaar is part of the GNU Project to produce a free operating system.

[openxs@chief bzr]$ bzr branch lp:percona-xtrabackup
You have not informed bzr of your Launchpad ID, and you must do this to
write to Launchpad or access private data. See "bzr help launchpad-login".
Branched 4983 revision(s).
[openxs@chief bzr]$ uname -a
Linux chief 2.6.35.14-106.fc14.x86_64 #1 SMP Wed Nov 23 13:07:52 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
[openxs@chief bzr]$ cd percona-xtrabackup/
[openxs@chief percona-xtrabackup]$ bzr version-info
revision-id: <email address hidden>
date: 2014-06-12 09:10:28 +0200
build-date: 2014-06-13 09:14:56 +0300
revno: 4983
branch-nick: percona-xtrabackup

[George Ormond Lorch III (gl-az)]

I apologize, my dyslexia kicked in, I meant 2.6.0 was where we saw it fixed. I believe I was on 2.2.something when I originally saw it about a year ago.

[Nirbhay Choubey (nirbhay)]

Hi George,
Not a problem. I got puzzled for a moment though. ;)

BTW, knielsen is getting this too with 2.6.0.

<knielsen> nirbhay: right, fresh repo branch, but still error: AssertionError: ('not present: %r', StaticTuple('', '', 'TREE_ROOT'))
...
<knielsen> nirbhay: I'm using debian stable
<knielsen> Bazaar (bzr) 2.6.0dev2

Also, the git repo seem to be a bit outdated (last updated Jan-13).
Will it be better to pick git repo instead?

[Valerii Kravchuk (valerii-kravchuk)]

Checked again, this time on Ubuntu 13.10 and with bzr 2.6.0. Still works for me:

openxs@ao756:~/bzr$ rm -rf percona-xtrabackup/
openxs@ao756:~/bzr$ cat /etc/issue
Ubuntu 13.10 \n \l

openxs@ao756:~/bzr$ bzr --version
Bazaar (bzr) 2.6.0
  Python interpreter: /usr/bin/python 2.7.5
  Python standard library: /usr/lib/python2.7
  Platform: Linux-3.11.0-19-generic-x86_64-with-Ubuntu-13.10-saucy
  bzrlib: /usr/lib/python2.7/dist-packages/bzrlib
  Bazaar configuration: /home/openxs/.bazaar
  Bazaar log file: /home/openxs/.bzr.log

Copyright 2005-2012 Canonical Ltd.
http://bazaar.canonical.com/

bzr comes with ABSOLUTELY NO WARRANTY. bzr is free software, and
you may use, modify and redistribute it under the terms of the GNU
General Public License version 2 or later.

Bazaar is part of the GNU Project to produce a free operating system.

openxs@ao756:~/bzr$ bzr branch lp:percona-xtrabackup
You have not informed bzr of your Launchpad ID, and you must do this to
write to Launchpad or access private data. See "bzr help launchpad-login".
Branched 4983 revisions.
openxs@ao756:~/bzr$ cd percona-xtrabackup/
openxs@ao756:~/bzr/percona-xtrabackup$ bzr version-info
revision-id: <email address hidden>
date: 2014-06-12 09:10:28 +0200
build-date: 2014-06-13 19:00:13 +0300
revno: 4983
branch-nick: percona-xtrabackup

[Sergei Glushchenko (sergei.glushchenko)]

I created bar shared repo, branched lp:percona-server, lp:percona-server/5.5 and lp:percona-xtrabackup. While branching the last one I got following:

$ bzr branch lp:percona-xtrabackup
bzr: ERROR: bzrlib.errors.BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing referenced chk root keys: [StaticTuple('sha1:3cbc15851d82ad932e73ba8b79db765e0e280b75',)].Run 'bzr reconcile --canonicalize-chks' on the affected repository.

Traceback (most recent call last):
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/commands.py", line 930, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/commands.py", line 1121, in run_bzr
    ret = run(*run_argv)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/commands.py", line 673, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/commands.py", line 697, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/cleanup.py", line 136, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/cleanup.py", line 166, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/builtins.py", line 1507, in run
    source_branch=br_from)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/bzrdir.py", line 367, in sprout
    create_tree_if_local=create_tree_if_local)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/cleanup.py", line 132, in run
    self.cleanups, self.func, self, *args, **kwargs)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/cleanup.py", line 166, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/bzrdir.py", line 417, in _sprout
    result_repo.fetch(source_repository, fetch_spec=fetch_spec)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/vf_repository.py", line 1268, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/decorators.py", line 218, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/vf_repository.py", line 2585, in fetch
    find_ghosts=find_ghosts)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/fetch.py", line 77, in __init__
    self.__fetch()
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/fetch.py", line 104, in __fetch
    self._fetch_everything_for_search(search_result)
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/fetch.py", line 132, in _fetch_everything_for_search
    stream, from_format, [])
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/vf_repository.py", line 2051, in insert_stream
    hint = self.target_repo.commit_write_group()
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/repository.py", line 633, in commit_write_group
    result = self._commit_write_group()
  File "/usr/local/Cellar/bazaar/2.6.0/libexec/bzrlib/repofmt/pack_repo.py", line 1727, in _commit_write_group
    hint = self._pack_collection._commit_write...

[Sergei Glushchenko (sergei.glushchenko)]

Upgrade from bar 2.6.0 to 2.7.0dev1 (current trunk) resolved the issue for me.

[Alexey Kopytov (akopytov)]

I was able to reproduce this on a test repository, but I'm not sure what to do about it. The suggested repair method ("bzr reconcile --canonicalize-chks") works on the repository level, rather than an individual branch and requires a _local_ repository.

So to me it looks like the repair should be done on LP servers. Reopening the bug for future references.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-689