Push fails with Revision {set([('null:',)])} not present in "KnitVersionedFiles

[Aaron Bentley]

Push of certain branches gives:
abentley@lappy:~/stacktest/bug294274.abentley$ bzr push bzr+ssh://localhost/home/abentley/stacktest/bug294274 -Derror
bzr: ERROR: bzrlib.errors.RevisionNotPresent: Revision {set([('null:',)])} not present in "KnitVersionedFiles(_KnitGraphIndex(CombinedGraphIndex(GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/fdf501cffc4e2b5390a98e9bbd68eab8.iix'), GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/a59c0f4d58ad729a14e5e1ef7fbbe2c9.iix'), GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/8aad7c12b8b3a00c27187cb176b08c4f.iix'), GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/6e69b1cc9fb22870858b38a9a9fe15d1.iix'), GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/3de2dd3e6e4e17e366457af7376211e2.iix'))), <bzrlib.knit._DirectPackAccess object at 0x9740dec>)".

Traceback (most recent call last):
  File "/home/abentley/bzr/bzr.dev/bzrlib/commands.py", line 893, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/home/abentley/bzr/bzr.dev/bzrlib/commands.py", line 839, in run_bzr
    ret = run(*run_argv)
  File "/home/abentley/bzr/bzr.dev/bzrlib/commands.py", line 539, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/abentley/bzr/bzr.dev/bzrlib/builtins.py", line 913, in run
    use_existing_dir=use_existing_dir)
  File "/home/abentley/bzr/bzr.dev/bzrlib/push.py", line 162, in _show_push_branch
    stop_revision=revision_id)
  File "/home/abentley/bzr/bzr.dev/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/home/abentley/bzr/bzr.dev/bzrlib/branch.py", line 1769, in push
    _override_hook_source_branch=_override_hook_source_branch)
  File "/home/abentley/bzr/bzr.dev/bzrlib/branch.py", line 2495, in _run_with_write_locked_target
    result = callable(*args, **kwargs)
  File "/home/abentley/bzr/bzr.dev/bzrlib/branch.py", line 1808, in _push_with_bound_branches
    result = self._basic_push(target, overwrite, stop_revision)
  File "/home/abentley/bzr/bzr.dev/bzrlib/branch.py", line 1831, in _basic_push
    graph=graph)
  File "/home/abentley/bzr/bzr.dev/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/home/abentley/bzr/bzr.dev/bzrlib/remote.py", line 1747, in update_revisions
    self.fetch(other, stop_revision)
  File "/home/abentley/bzr/bzr.dev/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/home/abentley/bzr/bzr.dev/bzrlib/branch.py", line 285, in fetch
    pb=nested_pb)
  File "/home/abentley/bzr/bzr.dev/bzrlib/remote.py", line 824, in fetch
    return inter.fetch(revision_id=revision_id, pb=pb, find_ghosts=find_ghosts)
  File "/home/abentley/bzr/bzr.dev/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/home/abentley/bzr/bzr.dev/bzrlib/repository.py", line 2811, in fetch
    pb, find_ghosts)
  File "/home/abentley/bzr/bzr.dev/bzrlib/fetch.py", line 110, in __init__
    self.__fetch()
  File "/home/abentley/bzr/bzr.dev/bzrlib/fetch.py", line 140, in __fetch
    self._fetch_everything_for_search(search, pp)
  File "/home/abentley/bzr/bzr.dev/bzrlib/fetch.py", line 166, in _fetch_everything_for_search
    for knit_kind, file_id, revisions in data_to_fetch:
  File "/home/abentley/bzr/bzr.dev/bzrlib/repository.py", line 1501, in item_keys_introduced_by
    file_ids = self.fileids_altered_by_revision_ids(revision_ids, inv_w)
  File "/home/abentley/bzr/bzr.dev/bzrlib/repository.py", line 1345, in fileids_altered_by_revision_ids
    selected_keys)
  File "/home/abentley/bzr/bzr.dev/bzrlib/repository.py", line 1321, in _find_file_ids_from_xml_inventory_lines
    line_iterator).iterkeys():
  File "/home/abentley/bzr/bzr.dev/bzrlib/repository.py", line 1261, in _find_text_key_references_from_xml_inventory_lines
    for line, line_key in line_iterator:
  File "/home/abentley/bzr/bzr.dev/bzrlib/knit.py", line 1454, in iter_lines_added_or_present_in_keys
    for line, key in source.iter_lines_added_or_present_in_keys(keys):
  File "/home/abentley/bzr/bzr.dev/bzrlib/knit.py", line 1461, in iter_lines_added_or_present_in_keys
    raise RevisionNotPresent(keys, repr(self))
RevisionNotPresent: Revision {set([('null:',)])} not present in "KnitVersionedFiles(_KnitGraphIndex(CombinedGraphIndex(GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/fdf501cffc4e2b5390a98e9bbd68eab8.iix'), GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/a59c0f4d58ad729a14e5e1ef7fbbe2c9.iix'), GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/8aad7c12b8b3a00c27187cb176b08c4f.iix'), GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/6e69b1cc9fb22870858b38a9a9fe15d1.iix'), GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/3de2dd3e6e4e17e366457af7376211e2.iix'))), <bzrlib.knit._DirectPackAccess object at 0x9740dec>)".

This is a smart-server problem. It does not exist in 1.8. Pushing with nosmart does not exhibit this problem.

So far, observed only in ubunet branches.

[Aaron Bentley (abentley)]

I've marked this private because the name "ubunet" is confidential.

[Aaron Bentley (abentley)]

Files to reproduce:
sftp://devpad/home/abentley/bug-295350.tar.bz2

[Guillermo Gonzalez (verterok)]

to workaround it I'm using bzr 1.8 for the moment

[Rick McBride (rmcbride)]

Definitely seeing this with bzr 1.9. Doesn't always happen on the first push of a branch, but almost always happens eventually.

[Elliot Murphy (statik)]

Is this bug will null revision the same as the bug where we get

bzr: ERROR: Revision {<email address hidden>} not present in "storagefs.py-20080815202844-sloqdzh6k56x32jy-1

when pushing a stacked branch?

[Elliot Murphy (statik)]

By the way, our team is hitting this bug 5 to 10 times a day, and lots of people end up having to delete and re-push branches to launchpad.

[Aaron Bentley (abentley)]

No, this bug is not Revision {<email address hidden>} not present in "storagefs.py-20080815202844-sloqdzh6k56x32jy-1. That bug is 288751.

Some of these comments appear to be intended for that one.

[Elliot Murphy (statik)]

Right, sorry about that, I got confused about that. Rick is still seeing this bug periodically, but bug 288751 is affecting us much more, and is the one that we are hitting several times a day. Thanks for the help, and thank you to everyone who is working on this bug. We really love using bzr.

[Martin Pool (mbp)]

I can confirm and reproduce this; it is similar to bug 288751.

There is some kind of problem in iter_lines_added_or_changed_in where it's called in the basis repository and cannot retrieve texts that in other respects seem to be stored correctly. This method currently knows about the internals of the kvf and perhaps it would be better to change it to get a record and return the lines from that...

[Martin Pool (mbp)]

Patch sent to list.

[Martin Pool (mbp)]

http://bundlebuggy.aaronbentley.com/project/bzr/request/%3Ce01316480811200433l1ed1082dr4f1e2c729e973cdb%40mail.gmail.com%3E

If you're getting bit by this, you could merge and test that bundle, which should fix 288751 too.

[Steve Alexander (stevea)]

Thanks Martin. I'll announce this patch in the online services stand-up meeting, an hour from now.

[Joshua Blount (jblount)]

Form what I can tell, the patch worked for me. Here are the four steps:

1. Push fails (after having pushed to LP 2 or 3 times previously)

joshuablount@r2d2:~/canonical/ubunet/fake-urls$ bzr push
Using saved push location: bzr+ssh://<email address hidden>/~joshuablount/ubunet/fake-urls
bzr: ERROR: Revision {<email address hidden>} not present in "urls.py-20080818162651-karc37pyo6dt1jye-3".

2. Patching dev version of bzr (which seems to partially fail)

joshuablount@r2d2:~/code/e/bzr.dev$ bzr patch ~/Desktop/20081120-295350-iter-changes-stacked.diff
1 out of 1 hunk FAILED -- saving rejects to file NEWS.rej
1 out of 11 hunks FAILED -- saving rejects to file bzrlib/repofmt/pack_repo.py.rej
patch unexpectedly ends in middle of line
bzr: ERROR: Patch application failed

3. Attempt to push with badly patched version of bzr

joshuablount@r2d2:~/canonical/ubunet/fake-urls$ /home/joshuablount/code/e/bzr.dev/bzr push lp:~joshuablount/ubunet/fake-urls
Pushed up to revision 253.

4. Profit

[Joshua Blount (jblount)]

Of course, I meant to comment on Bug 288751 , and there's no way to delete these comments.

[James Henstridge (jamesh)]

I had a branch giving the "Revision {set([('null:',)])} not present ..." error with bzr 1.9-1~bazaar1~intrepid1. It is a "1.6" format branch, if that helps.

I just tried using bzr.dev with your patch applied, and successfully pushed to the branch. It looks like Launchpad successfully scanned the change I pushed, so it looks like it worked.

[Martin Pool (mbp)]

This has been sent to pqm and should be in bzr.dev soon, and then in statik's nightly builds. Please let me know how it goes for you.

[dobey (dobey)]

OK. So I just got this bug TWICE with a patched bzr. There was a change in trunk which removed a directory, and when I tried to push to two separate branches, after merging with that trunk, I hit this bug.

[Andrew Bennetts (spiv)]

Rodney: are those branches publically accessible?

[dobey (dobey)]

Andrew: No, they are private branches.

On Mon, 2008-11-24 at 00:50 +0000, Andrew Bennetts wrote:
> Rodney: are those branches publically accessible?
>
> --
> Push fails with Revision {set([('null:',)])} not present in "KnitVersionedFiles
> https://bugs.launchpad.net/bugs/295350
> You received this bug notification because you are a member of Ubunet
> hackers, which is a direct subscriber.
>
> Status in Bazaar Version Control System: Confirmed
>
> Bug description:
> Push of certain branches gives:
> abentley@lappy:~/stacktest/bug294274.abentley$ bzr push bzr+ssh://localhost/home/abentley/stacktest/bug294274 -Derror
> bzr: ERROR: bzrlib.errors.RevisionNotPresent: Revision {set([('null:',)])} not present in "KnitVersionedFiles(_KnitGraphIndex(CombinedGraphIndex(GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/fdf501cffc4e2b5390a98e9bbd68eab8.iix'), GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/a59c0f4d58ad729a14e5e1ef7fbbe2c9.iix'), GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/8aad7c12b8b3a00c27187cb176b08c4f.iix'), GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/6e69b1cc9fb22870858b38a9a9fe15d1.iix'), GraphIndex('file:///home/abentley/push-probs/ubunet/.bzr/repository/indices/3de2dd3e6e4e17e366457af7376211e2.iix'))), <bzrlib.knit._DirectPackAccess object at 0x9740dec>)".
>
> Traceback (most recent call last):
> File "/home/abentley/bzr/bzr.dev/bzrlib/commands.py", line 893, in run_bzr_catch_errors
> return run_bzr(argv)
> File "/home/abentley/bzr/bzr.dev/bzrlib/commands.py", line 839, in run_bzr
> ret = run(*run_argv)
> File "/home/abentley/bzr/bzr.dev/bzrlib/commands.py", line 539, in run_argv_aliases
> return self.run(**all_cmd_args)
> File "/home/abentley/bzr/bzr.dev/bzrlib/builtins.py", line 913, in run
> use_existing_dir=use_existing_dir)
> File "/home/abentley/bzr/bzr.dev/bzrlib/push.py", line 162, in _show_push_branch
> stop_revision=revision_id)
> File "/home/abentley/bzr/bzr.dev/bzrlib/decorators.py", line 138, in read_locked
> result = unbound(self, *args, **kwargs)
> File "/home/abentley/bzr/bzr.dev/bzrlib/branch.py", line 1769, in push
> _override_hook_source_branch=_override_hook_source_branch)
> File "/home/abentley/bzr/bzr.dev/bzrlib/branch.py", line 2495, in _run_with_write_locked_target
> result = callable(*args, **kwargs)
> File "/home/abentley/bzr/bzr.dev/bzrlib/branch.py", line 1808, in _push_with_bound_branches
> result = self._basic_push(target, overwrite, stop_revision)
> File "/home/abentley/bzr/bzr.dev/bzrlib/branch.py", line 1831, in _basic_push
> graph=graph)
> File "/home/abentley/bzr/bzr.dev/bzrlib/decorators.py", line 192, in write_locked
> result = unbound(self, *args, **kwargs)
> File "/home/abentley/bzr/bzr.dev/bzrlib/remote.py", line 1747, in update_revisions
> self.fetch(other, stop_revision)
> File "/home/abentley/bzr/bzr.dev/bzrlib/decorators.py", line 192, in write_locked
> result = unbound(self, *args, **kwargs)
> File "/home/abentley/bzr/bzr.dev/bzrlib/branch.py", line 285, in fetch
> pb=nested_pb)
> ...

[dobey (dobey)]

Here is the ~/.bzr.log of a push in question. This is with Martin's patch applied. I also seem to be unable to bzr branch this branch now. When I try, bzr seems to just sit there forever, slowly leaking memory. However, merging those branches into another seems to work ok still. Should I file a separate bug for that?

0.062 bzr arguments: [u'push']
0.063 looking for plugins in /home/dobey/.bazaar/plugins
0.115 looking for plugins in /usr/lib/python2.5/site-packages/bzrlib/plugins
0.116 Plugin name gtk already loaded
0.282 encoding stdout as sys.stdout encoding 'UTF-8'
1.161 ssh implementation is OpenSSH
13.530 fetch up to rev {$replaced-revid}
20.105 Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 893, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 839, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 539, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 913, in run
    use_existing_dir=use_existing_dir)
  File "/usr/lib/python2.5/site-packages/bzrlib/push.py", line 162, in _show_push_branch
    stop_revision=revision_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 1769, in push
    _override_hook_source_branch=_override_hook_source_branch)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 2495, in _run_with_write_locked_target
    result = callable(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 1808, in _push_with_bound_branches
    result = self._basic_push(target, overwrite, stop_revision)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 1831, in _basic_push
    graph=graph)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1747, in update_revisions
    self.fetch(other, stop_revision)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 285, in fetch
    pb=nested_pb)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 824, in fetch
    return inter.fetch(revision_id=revision_id, pb=pb, find_ghosts=find_ghosts)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2814, in fetch
    pb, find_ghosts)
  File "/usr/lib/python2.5/site-packages/bzrlib/fetch.py", line 110, in __init__
    self.__fetch()
  File "/usr/lib/python2.5/site-packages/bzrlib/fetch.py", line 140, in __fetch
    self._fetch_everything_for_search(search, pp)
  File "/usr/lib/python2.5/site-packages/bzrlib/fetch.py", line 166, in _fetch_everything_for_sea...

[Ivan Zorin (iaz)]

Confirm. I use bzr in intrepid
> bzr --version
Bazaar (bzr) 1.13dev
  Python interpreter: /usr/bin/python 2.5.2
  Python standard library: /usr/lib/python2.5
...
and in jaunty (where python is 2.5.4); but in both systems for "bzr push" I get almost exactly error message as in this bug report.
This is very good, that developers know about this problem, but, however, as far as I understand, I can't upload(push) branches using this version bazaar, can I?

[Robert Collins (lifeless)]

This was fixed in bzr 1.10