Another case of "checksum mismatch"

[Lalo Martins]

I'm encountering another “checksum mismatch” issue that may or may not be related to this one, but is certainly not related to the one I had before. Error message:

bzr: ERROR: checksum mismatch: 'c5cdb6209aac6038ac50f695fd6796d6' != '4fe50dc7365f7b45a45967f4ff488f01' in branches/news_and_event_publishing:44092
zsh: exit 3 bzr up

Log section:
Thu 2010-12-23 20:01:23 +0100
0.040 bazaar version: 2.2.0
0.040 bzr arguments: [u'up']
0.051 looking for plugins in /home/lalo/.bazaar/plugins
0.084 looking for plugins in /usr/lib/python2.7/site-packages/bzrlib/plugins
0.090 encoding stdout as sys.stdout encoding 'UTF-8'
0.125 opening working tree '/repo/home/lalo/site/news_and_event_publishing'
0.157 bzr-svn: using Subversion 1.6.13 ()
0.456 Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/bzrlib/commands.py", line 911, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/bzrlib/commands.py", line 1111, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.7/site-packages/bzrlib/commands.py", line 689, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.7/site-packages/bzrlib/commands.py", line 704, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/bzrlib/builtins.py", line 1421, in run
    old_tip = branch.update(possible_transports)
  File "/usr/lib/python2.7/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/bzrlib/branch.py", line 2761, in update
    self.pull(master, overwrite=True)
  File "/usr/lib/python2.7/site-packages/bzrlib/branch.py", line 1068, in pull
    possible_transports=possible_transports, *args, **kwargs)
  File "/home/lalo/.bazaar/plugins/svn/branch.py", line 817, in pull
    stop_revision, overwrite, limit=limit)
  File "/home/lalo/.bazaar/plugins/svn/branch.py", line 749, in _update_revisions
    project=self.source.project, mapping=self.source.mapping)
  File "/home/lalo/.bazaar/plugins/svn/fetch.py", line 1453, in fetch
    pack_hint = self._fetch_revisions(needed, pb)
  File "/home/lalo/.bazaar/plugins/svn/fetch.py", line 1380, in _fetch_revisions
    use_replay=self._use_replay)
  File "/home/lalo/.bazaar/plugins/svn/fetch.py", line 1350, in _fetch_revisions_nochunks
    parent_revmeta)
  File "/home/lalo/.bazaar/plugins/svn/fetch.py", line 1292, in _fetch_revision_switch
    report_inventory_contents(reporter, parent_revnum, start_empty)
  File "/home/lalo/.bazaar/plugins/svn/errors.py", line 144, in convert
    return unbound(*args, **kwargs)
  File "/home/lalo/.bazaar/plugins/svn/fetch.py", line 1030, in report_inventory_contents
    reporter.finish()
  File "/home/lalo/.bazaar/plugins/svn/transport.py", line 252, in update
    if self._last_progress > progress:
  File "/home/lalo/.bazaar/plugins/svn/transport.py", line 252, in update
    if self._last_progress > progress:
  File "/home/lalo/.bazaar/plugins/svn/transport.py", line 252, in update
    if self._last_progress > progress:
  File "/home/lalo/.bazaar/plugins/svn/fetch.py", line 419, in apply_textdelta
    return self._apply_textdelta(base_checksum)
  File "/home/lalo/.bazaar/plugins/svn/fetch.py", line 627, in _apply_textdelta
    self.editor.revmeta.revnum)
TextChecksumMismatch: checksum mismatch: 'c5cdb6209aac6038ac50f695fd6796d6' != '4fe50dc7365f7b45a45967f4ff488f01' in branches/news_and_event_publishing:44092

0.456 Transferred: 14kB (46.3kB/s r:0kB w:0kB u:14kB)
0.456 return code 3

What I believe happens is that two of the revisions committed by other developers today weren't committed in the “root” of the project source tree, and so bzr-svn/subvertpy doesn't see one or both of those revisions as being part of the ancestry. Looking at the commit on trac, it gives the “location” as branches/news_and_event_publishing/ableton, while most other commits show as branches/news_and_event_publishing.

I committed revision 44073, using bzr-svn, then my colleague committed 44074 which had the trailing path in the location, then I tried to “bzr up” and it said I was already up to date, at which point I started expecting trouble later. Then I committed 44075-79 and 44091, which were fine because no files conflicted with '74, then he committed 44092 and now I can neither update or commit.

I tried this with 1.0.4 (rel), trunk (branch), and 1.1 (branch), because I remembered something about there being already a fix for this, but although the symptoms look similar, it's apparently a rather different problem.

[Jelmer Vernooij (jelmer)]

On Thu, 2010-12-23 at 19:21 +0000, Lalo Martins wrote:
> Public bug reported:
>
> I'm encountering another “checksum mismatch” issue that may or may not
> be related to this one, but is certainly not related to the one I had
> before. Error message:
>
> bzr: ERROR: checksum mismatch: 'c5cdb6209aac6038ac50f695fd6796d6' != '4fe50dc7365f7b45a45967f4ff488f01' in branches/news_and_event_publishing:44092
> zsh: exit 3 bzr up
Have you tried this in a clean branch? The main issue with this kind of
error is broken data in the existing repository.

Cheers,

Jelmer

[Lalo Martins (lalo.martins)]

Yes. I tried making a new checkout of the same branch (outside the shared repository), that didn't help (failed when it reached the same revision). I didn't remember to delete my bzr-svn cache though, I'll try in the morning and let you know.

[Lalo Martins (lalo.martins)]

Nope, that didn't help. Same end results, but with a lot of “inconsistent details in skipped record” before the final fail. Sample one:

inconsistent details in skipped record: ('43373@96f830f6-37e5-0310-8a40-84625fa2d7b7:branches%2Fnews_and_event_publishing%2Fableton%2Fnext%2Fcontrollers%2Fadmin%2Fcontent%2Ftemplates', 'svn-v4:96f830f6-37e5-0310-8a40-84625fa2d7b7:branches/news_and_event_publishing:43373') ('1778098 23 0 0', ((),)) ('9570474 23 0 0', ((('43373@96f830f6-37e5-0310-8a40-84625fa2d7b7:branches%2Fnews_and_event_publishing%2Fableton%2Fnext%2Fcontrollers%2Fadmin%2Fcontent%2Ftemplates', 'svn-v4:96f830f6-37e5-0310-8a40-84625fa2d7b7:branches/news_and_event_publishing:43373'),),))

I'd rather not attach the log file, as it's a bit too revealing about our source tree (and while I don't believe in security through obscurity, my employers certainly do), but I can mail it privately if that would help.

Also I'll try to reproduce it in a fresh, sample svn repository that has no confidential code and doesn't take me half an hour to check out :-)

[Jelmer Vernooij (jelmer)]

It'd be great to have a reproducible version of this.

[Lalo Martins (lalo.martins)]

Sorry, with holidays and all I didn't have a chance to set up a new repository. But I do believe this will reproduce it:

- a repository with a trunk and stuff in it, in particular, subdirectories
- create branch /branches/foo
- check foo out with svn and with bzr
- do one commit with bzr; might not be necessary, but just to make sure bzr has its own idea of the contents
- in svn, commit from inside a subdirectory that is version controlled; eg: cd svn/foo/bar; touch baz; svn ci -m "this may break bzr"
- bzr up; this will tell you your tree is up to date
- do another commit from bzr; again, may not be necessary
- bzr up; this will either break, in which case QED, or tell you the tree is up to date, in which case proceed to:
- make a commit using svn, from the branch root, but NOT what was changed in the previous (foo/bar/baz) revision: cd svn/foo; touch something; svn ci -m "bzr will see this one"
- bzr up, now it's definitely lost

I'll do those experiments tomorrow, if you don't beat me to it, and post the results, and possibly a tar.bz2 of the repository.

[Lalo Martins (lalo.martins)]

nope, couldn't reproduce it :-( when I get to step 6 above, bzr does see my svn commit. I have tried all variations of committing on a subdir that I could think of (including checking out just the subdir with svn) and nothing reproduces this. So it might be unrelated to what I thought. Sorry, I'm at a loss here.

I guess it would be fair to close this as invalid (can't be reproduced), but meanwhile the actual branch where this happened is still inaccessible for me with bzr, which forces me to work with svn for that branch only. (FWIW, last time it happened, when the branch was re-merged to trunk that didn't break trunk. I may try merging trunk *into* the branch to see if that fixes it.)

[Lalo Martins (lalo.martins)]

for the record, merging trunk doesn't fix it either.

[Jelmer Vernooij (jelmer)]

On Wed, 2011-01-05 at 16:46 +0000, Lalo Martins wrote:
> nope, couldn't reproduce it :-( when I get to step 6 above, bzr does see
> my svn commit. I have tried all variations of committing on a subdir
> that I could think of (including checking out just the subdir with svn)
> and nothing reproduces this. So it might be unrelated to what I thought.
> Sorry, I'm at a loss here.
>
> I guess it would be fair to close this as invalid (can't be reproduced),
> but meanwhile the actual branch where this happened is still
> inaccessible for me with bzr, which forces me to work with svn for that
> branch only. (FWIW, last time it happened, when the branch was re-merged
> to trunk that didn't break trunk. I may try merging trunk *into* the
> branch to see if that fixes it.)
Can you try to unset all bzr: revision properties and see if a clean
clone works then?

Cheers,

Jelmer

[Lalo Martins (lalo.martins)]

sorry... delete the properties where exactly? Every revision in the branch? The revision that has the checksum mismatch doesn't have any bzr: properties (it was Mike's svn commit)

Is there an easier way to do this that I'm missing or do I have to go through the log by hand and find out what to delete and from where?

[Jelmer Vernooij (jelmer)]

Hi Lalo,

Sorry for the delayed answer.

On Fri, 2011-01-07 at 16:18 +0000, Lalo Martins wrote:
> sorry... delete the properties where exactly? Every revision in the
> branch? The revision that has the checksum mismatch doesn't have any
> bzr: properties (it was Mike's svn commit)
Any revision in the revision ancestry of the problematic commit that has
bzr: properties (file properties or revision properties).

> Is there an easier way to do this that I'm missing or do I have to go
> through the log by hand and find out what to delete and from where?
I'm afraid that's the only way at the moment. :-( It should be possible
to automate it using a simply subvertpy script if the revisions are in
revision properties.

Otherwise, it should be possible to use some regular expression magic.

Cheers,

Jelmer

[Philip Peitsch (philip-peitsch)]

In case anyone is interested, I've attached an xslt that can be used with svn log outputs to generate a script that can clear bzr meta data entries.

Use the output from the following command:

svn log --xml --with-all-revprops <subversion repository> > allsvn.xml

Then a tool (e.g., xmlstarlet) to transform the xml using the attached xslt. This can then be dumped into a bat file or shell script and executed to mass delete the bzr metadata.