ErrorFromSmartServer: Absent factory for StaticTuple

[Sean M. Pappalardo]

Branch lp:mixxx/1.9 recently (in the last few weeks) has begin crashing bzr when people make new local checkouts or branches with this error:
bzr: ERROR: bzrlib.errors.ErrorFromSmartServer: Error received from smart server: ('error', "Absent factory for StaticTuple('knob_rotary_s20.png-20100321152943-5b6m28lw59aoyi01-107', 'jus@local-20100321153201-ijr7fnx2lljnntyg')")

...using v2.1.2.

bzr check also crashed with: bzr: ERROR: exceptions.KeyError: ('audiotagger.cpp-20101031134221-wol9iyxr3fxcehpg-1', 'raffitea-20101031205517-v30qwekt2fc8cqq2')

I'm running bzr pack right now (I have v2.1.1) so hopefully that fixes the issue, but I was told to report this here anyway per 'lifeless' in #launchpad on IRC who also noticed that the branch is stacked on "" and the missing data is in r1!

(BTW, should release branches like this one be stacked at all?)

[Martin Pool (mbp)]

This looks a bit like bug 599670, which expired incomplete.

I can reproduce this branching from there to a brand new standalone branch.

Traceback:
 Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 936, in exception_to_return_code
     return the_callable(*args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 1136, in run_bzr
     ret = run(*run_argv)
   File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 697, in run_argv_aliases
     return self.run(**all_cmd_args)
   File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 719, in run
     return self._operation.run_simple(*args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 135, in run_simple
     self.cleanups, self.func, *args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
     result = func(*args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/bzrlib/builtins.py", line 1296, in run
     source_branch=br_from)
   File "/usr/lib/python2.7/dist-packages/bzrlib/bzrdir.py", line 451, in sprout
     create_tree_if_local=create_tree_if_local)
   File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 131, in run
     self.cleanups, self.func, self, *args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
     result = func(*args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/bzrlib/bzrdir.py", line 492, in _sprout
     result_repo.fetch(source_repository, fetch_spec=fetch_spec)
   File "/usr/lib/python2.7/dist-packages/bzrlib/repository.py", line 1783, in fetch
     find_ghosts=find_ghosts, fetch_spec=fetch_spec)
   File "/usr/lib/python2.7/dist-packages/bzrlib/decorators.py", line 208, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/bzrlib/repository.py", line 3390, in fetch
     find_ghosts=find_ghosts)
   File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 75, in __init__
     self.__fetch()
   File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 102, in __fetch
     self._fetch_everything_for_search(search_result)
   File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 130, in _fetch_everything_for_search
     stream, from_format, [])
   File "/usr/lib/python2.7/dist-packages/bzrlib/repository.py", line 4065, in insert_stream
     src_format, is_resume)
   File "/usr/lib/python2.7/dist-packages/bzrlib/repository.py", line 4143, in insert_stream_without_locking
     self.target_repo.chk_bytes.insert_record_stream(substream)
   File "/usr/lib/python2.7/dist-packages/bzrlib/groupcompress.py", line 1623, in insert_record_stream
     for _ in self._insert_record_stream(stream, random_id=False):
   File "/usr/lib/python2.7/dist-packages/bzrlib/groupcompress.py", line 1689, in _insert_record_stream
     for record in stream:
   File "/usr/lib/python2.7/dist-packages/bzrlib/smart/repository.py", line 607, in wrap_and_count
     for record in substream.read():
   File "/usr/lib/python2.7/dist-packages/bzrlib/versionedfile.py", line 1816, in read
     for ...

[Andrew Bennetts (spiv)]

FWIW, branching from nosmart+lp:… triggers the same error locally (rather than server-side). So it's not a problem caused by reading the repository in isolation from the designated fallback(s).

[Sean M. Pappalardo (pegasus-renegadetech)]

The bzr pack is finished and doesn't seem to have helped because attached is what happens when I try to bzr check the branch afterwords.

(Oh, it seems I'm on bzr 2.1.2 as well. (Debian Squeeze.))

[RJ Skerry-Ryan (rryan)]

I hate to ping this bug but is there any possible workaround we can suggest to our developers? We're a fairly active project and because of this bug new contributors and users cannot check out our source via launchpad.

[Andrew Bennetts (spiv)]

Here's a workaround:

 * Install this plugin: “bzr branch lp:~spiv/+junk/bzr-fetch-all-records ~/.bazaar/fetch_all_records”
 * Run “bzr fetch-all-records -d lp:mixxx/1.9 lp:mixxx”

It appears that all the records bzr can't find are present in lp:mixxx, so that plugin is just a quick hack that indiscriminately copies *every* record not already in the target repository from the source repository. This will undoubtedly be grabbing a bunch of irrelevant stuff (about 25M worth in my quick test), but it also fills in the records that are missing. Clients that branch from this repo will happily skip over the excess records and just fetch what is relevant, so it shouldn't cause any lasting bloat issues.

I'm pushing up a copy of lp:mixxx/1.9 repaired in this manner to lp:~spiv/mixxx/1.9-repaired.

You may need to apply the workaround to local copies with these symptoms, as well as repairing the copy on Launchpad.

[Andrew Bennetts (spiv)]

I've bundled up that command (fetch-all-records) along with a few others that I've accumulated for investigating similar issues in the past in a new plugin: lp:bzr-repodebug.

[Sean M. Pappalardo (pegasus-renegadetech)]

Will checkouts suffer bloat with this workaround? Many Mixxx devs and users use local checkouts instead of branches.

[Sean M. Pappalardo (pegasus-renegadetech)]

Also, after checking out repodebug, when I try to use the command (even doing just bzr help fetch-all-records) I get:
bzr: ERROR: exceptions.KeyError: 'directory'

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 853, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1055, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 631, in run_argv_aliases
    args, opts = parse_args(self, argv, alias_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 748, in parse_args
    parser = option.get_optparser(command.options())
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 614, in options
    o = option.Option.OPTIONS[o]
KeyError: 'directory'

bzr 2.1.2 on python 2.6.6 (Linux-2.6.33.7.2-rt30-1-686-i686-with-debian-6.0.1)
arguments: ['/usr/bin/bzr', 'fetch-all-records', '-d', 'lp:mixxx/1.9', 'lp:mixxx']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bzrtools /usr/lib/python2.6/dist-packages/bzrlib/plugins/bzrtools [2.1.0]
  cia /usr/lib/python2.6/dist-packages/bzrlib/plugins/cia [1.0.0dev]
  dbus /usr/lib/python2.6/dist-packages/bzrlib/plugins/dbus [0.1.0dev]
  extmerge /home/pegasus/.bazaar/plugins/extmerge [unknown]
  gtk /usr/lib/python2.6/dist-packages/bzrlib/plugins/gtk [0.99.1]
  launchpad /usr/lib/python2.6/dist-packages/bzrlib/plugins/launchpad [2.1.2]
  netrc_credential_store /usr/lib/python2.6/dist-packages/bzrlib/plugins/netrc_credential_store [2.1.2]
  news_merge /usr/lib/python2.6/dist-packages/bzrlib/plugins/news_merge [2.1.2]
  repodebug /home/pegasus/.bazaar/plugins/repodebug [unknown]

[Andrew Bennetts (spiv)]

fetch-all-records will add some bloat to a repository, but bzr won't copy that bloat elsewhere. So repairing a local branch/checkout with it will add bloat. Making a new branch/checkout from a repaired copy with have no bloat.

As for your problem with the repodebug plugin: that's due to using an older bzr. It definitely works with 2.3 and probably 2.2 as well.

[Sean M. Pappalardo (pegasus-renegadetech)]

Upgraded bzr and now I get a TooManyConcurrentRequests error when trying to --dry-run your suggested command.

[Sean M. Pappalardo (pegasus-renegadetech)]

Ok, fetch-all-records fixed our problem. Thank you!

[John A Meinel (jameinel)]

Yay!

I investigated a bit further, and it looks like a bug with how this branch was
stacked. Specifically:

1) The current lp:lightdm branch has this in branch.conf:
stacked_on_location = ""
parent_location = ../../../~robert-ancell/lightdm/trunk/

That hints that it used to be a stacked branch, and it used to be stacked on robert-ancell's branch.

2) I investigated, and ~robert-ancell/lightdm/trunk *does* have the text revision that was missing in lp:lightdm. Though potentially you filled it in manually, and I got to the party late.

3) The missing text is from a fairly old revision (from 2010).

So my guess is that the text was present in ~robert-ancell and the ~lightdm-team branch was stacked on it, so it didn't copy that text. When the ~lightdm-team branch was later promoted to development focus and then unstacked, it didn't get all of the history filled in properly.

4) I grabbed a copy of "lp:lightdm" while it was broken using hitchhiker. I was unable to branch it to another location. If I edited ".bzr/branch/branch.conf" to change it to:
stacked_on_location = "bzr+ssh://bazaar.launchpad.net/~robert-ancell/lightdm/trunk"

I was then able to "bzr branch lightdm xxx"

So my guess is that the branch was unstacked by just removing the
stacking_location, rather than "bzr reconfigure --unstacked". Which leaves us
with a branch that is missing some of its history, and doesn't have a reference
for where to find that history.

[John A Meinel (jameinel)]

@sean: "Will checkouts suffer bloat with this workaround? Many Mixxx devs and users use local checkouts instead of branches."

No. This adds more data to the repository, but it doesn't change what data will be in a checkout (heavy or lightweight).

[Robert Ancell (robert-ancell)]

The lightdm bug is in bug 816296. I haven't done anything special with it afaik, other than to bzr push to the new location and bzr pull --remember. Still not fixed, and I don't know what to run to fix it.