KnitCorrupt: corrupt: incorrect number of lines while branching from svn

[John Steele Scott]

I ran the following set of commands to get the backtrace below. $SVNROOT here is an svn+ssh URL.

This is similar to bug #160081, but that is with a much older version, and does not appear to involve the svn plugin.

jscott@jscott-laptop:~/src$ bzr init-repo --rich-root-pack bzr-repo
jscott@jscott-laptop:~/src$ cd bzr-repo/
jscott@jscott-laptop:~/src/bzr-repo$ time bzr branch $SVNROOT/trunk trunk
bzr: ERROR: bzrlib.errors.KnitCorrupt: Knit <bzrlib.knit._PackAccess object at 0x157b850> corrupt: incorrect number of lines 30 != 29 for version {svn-v3-trunk0:85072ce8-75ea-0310-bde9-cb3a2c948a9f:trunk:8410}

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 834, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 790, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 927, in run
    hardlink=hardlink)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/remote.py", line 76, in sprout
    result_repo.fetch(repo, revision_id=revision_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 949, in fetch
    return inter.fetch(revision_id=revision_id, pb=pb, find_ghosts=find_ghosts)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/fetch.py", line 704, in fetch
    self._fetch_switch(needed, pb, lhs_parent)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/fetch.py", line 661, in _fetch_switch
    reporter.finish_report(pool)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/errors.py", line 117, in convert
    return unbound(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/transport.py", line 280, in finish_report
    self._baton, pool)
  File "/var/lib/python-support/python2.5/libsvn/ra.py", line 740, in svn_ra_reporter2_invoke_finish_report
    return apply(_ra.svn_ra_reporter2_invoke_finish_report, args)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/fetch.py", line 471, in abort_edit
    self.target.abort_write_group()
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 482, in abort_write_group
    self._abort_write_group()
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 1850, in _abort_write_group
    self._pack_collection._abort_write_group()
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 1654, in _abort_write_group
    self._new_pack.abort()
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 318, in abort
    self.write_stream.close()
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/__init__.py", line 281, in close
    del _file_streams[self.transport.abspath(self.relpath)]
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/local.py", line 103, in abspath
    self._local_base, urlutils.unescape(relpath)))
  File "/usr/lib/python2.5/site-packages/bzrlib/urlutils.py", line 525, in unescape
    unquoted = urllib.unquote(url)
  File "urllib.py", line 1154, in unquote
    for i in xrange(1, len(res)):
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/fetch.py", line 400, in close_edit
    self._finish_commit()
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/fetch.py", line 463, in _finish_commit
    self.target.add_revision(self.revid, rev, self.inventory)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 554, in add_revision
    self.add_inventory(revision_id, inv, rev.parent_ids)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 516, in add_inventory
    inv_lines, check_content=False)
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/knitrepo.py", line 111, in _inventory_add_lines
    check_content=check_content)[0]
  File "/usr/lib/python2.5/site-packages/bzrlib/versionedfile.py", line 138, in add_lines_with_ghosts
    parent_texts, nostore_sha, random_id, check_content)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 984, in _add_lines_with_ghosts
    parent_texts, None, nostore_sha, random_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1060, in _add
    left_matching_blocks)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 915, in _merge_annotations
    reference_content = self._get_content(parents[0], parent_texts)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 972, in _get_content
    text_map, contents_map = self._get_content_maps([version_id])
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1156, in _get_content_maps
    record_map = self._get_record_map(version_ids)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1121, in _get_record_map
    self._data.read_records_iter(records):
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 2639, in read_records_iter
    content, digest = self._parse_record(version_id, data)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 2555, in _parse_record
    version_id))
KnitCorrupt: Knit <bzrlib.knit._PackAccess object at 0x157b850> corrupt: incorrect number of lines 30 != 29 for version {svn-v3-trunk0:85072ce8-75ea-0310-bde9-cb3a2c948a9f:trunk:8410}

bzr 1.3.1rc1 on python 2.5.1.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'branch', 'svn+ssh://moon/usr/local/svn/rocksoft/src/trunk', 'trunk']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_AU.UTF-8'
plugins:
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
  svn /usr/lib/python2.5/site-packages/bzrlib/plugins/svn [0.4.9]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

real 37m59.789s
user 22m23.956s
sys 0m54.227s

[John Steele Scott (toojays)]

I get a the same error with an up-to-date checkout of bzr-1.4 branch and bzr-svn 0.4 branch:

jscott@jscott-laptop:~/src/b$ time bzr14 branch $SVNROOT/trunk trunk
Initialising Subversion metadata cache in /home/jscott/.bazaar/svn-cache-exp/85072ce8-75ea-0310-bde9-cb3a2c948a9f
using experimental bzr-svn mappings; output may change between revisions
bzr: ERROR: bzrlib.errors.KnitCorrupt: Knit <bzrlib.knit._PackAccess object at 0xa753cd0> corrupt: incorrect number of lines 41 != 40 for version {svn-v3-trunk0:85072ce8-75ea-0310-bde9-cb3a2c948a9f:trunk:8410}

Traceback (most recent call last):
  File "/home/jscott/src/bzr/bzr.1.4/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/home/jscott/src/bzr/bzr.1.4/bzrlib/commands.py", line 802, in run_bzr
    ret = run(*run_argv)
  File "/home/jscott/src/bzr/bzr.1.4/bzrlib/commands.py", line 504, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/jscott/src/bzr/bzr.1.4/bzrlib/builtins.py", line 954, in run
    hardlink=hardlink)
  File "/home/jscott/src/bzr-svn/0.4/remote.py", line 76, in sprout
    result_repo.fetch(repo, revision_id=revision_id)
  File "/home/jscott/src/bzr/bzr.1.4/bzrlib/repository.py", line 947, in fetch
    return inter.fetch(revision_id=revision_id, pb=pb, find_ghosts=find_ghosts)
  File "/home/jscott/.bazaar/plugins/svn/fetch.py", line 741, in fetch
    self._fetch_switch(needed, pb)
  File "/home/jscott/.bazaar/plugins/svn/fetch.py", line 700, in _fetch_switch
    reporter.finish_report(pool)
  File "/home/jscott/src/bzr-svn/0.4/errors.py", line 117, in convert
    return unbound(*args, **kwargs)
  File "/home/jscott/src/bzr-svn/0.4/transport.py", line 264, in finish_report
    self._baton, pool)
  File "/var/lib/python-support/python2.5/libsvn/ra.py", line 740, in svn_ra_reporter2_invoke_finish_report
    return apply(_ra.svn_ra_reporter2_invoke_finish_report, args)
  File "/home/jscott/.bazaar/plugins/svn/fetch.py", line 488, in abort_edit
    self.target.abort_write_group()
  File "/home/jscott/src/bzr/bzr.1.4/bzrlib/repository.py", line 479, in abort_write_group
    self._abort_write_group()
  File "/home/jscott/src/bzr/bzr.1.4/bzrlib/repofmt/pack_repo.py", line 1849, in _abort_write_group
    self._pack_collection._abort_write_group()
  File "/home/jscott/src/bzr/bzr.1.4/bzrlib/repofmt/pack_repo.py", line 1653, in _abort_write_group
    self._new_pack.abort()
  File "/home/jscott/src/bzr/bzr.1.4/bzrlib/repofmt/pack_repo.py", line 318, in abort
    self.write_stream.close()
  File "/home/jscott/src/bzr/bzr.1.4/bzrlib/transport/__init__.py", line 273, in close
    del _file_streams[self.transport.abspath(self.relpath)]
  File "/home/jscott/src/bzr/bzr.1.4/bzrlib/transport/local.py", line 103, in abspath
    self._local_base, urlutils.unescape(relpath)))
  File "/home/jscott/src/bzr/bzr.1.4/bzrlib/urlutils.py", line 525, in unescape
    unquoted = urllib.unquote(url)
  File "urllib.py", line 1154, in unquote
    for i in xrange(1, len(res)):
  File "/home/jscott/.bazaar/plugins/svn/fetch.py", line 418, in close_edit
    self._finish_commit()
  File "/home/jscott/.bazaar/plugins/svn/fetch.py", line 480, in _finish_commit
 ...

[Andrew Bennetts (spiv)]

Is there a filename with a newline in it? If so, this is a duplicate of bug 3918.

[John Steele Scott (toojays)]

I had a look at revision 8410 in my SVN repo (and at revisions on either side), and there are no files with newlines in the name. Unless the tool I'm using to look in the repo doesn't show the newlines, of course.

[Andrew Straw (astraw)]

I'm getting what appears to be the same bug. (See attached log). I also don't have files with newlines, and that particular svn commit didn't include anything particularly weird looking. I can send the output of "svn diff -r1271:1272" if that would be useful.

This is with bzr 1.4rc2 (tarball) and bzr-svn from the 0.4 branch revo 1096.

I'm running this command on an Ubuntu Gutsy machine with subversion and python-subversion version 1.4.4dfsg1-1ubuntu3.

[John Steele Scott (toojays)]

Andrew,

I'm not a bzr developer, but it might be helpful for them if you could post the output you get from the assert with the below patch applied. I can't do it because I'm working with a proprietary repo. It looked to me like the corrupt "knit" wasn't really a knit at all. AFAIK a knit resembles a diff, and what was coming out here looked a lot more like an inventory.

=== modified file 'bzrlib/knit.py'
--- bzrlib/knit.py 2008-04-09 23:35:55 +0000
+++ bzrlib/knit.py 2008-04-24 23:48:42 +0000
@@ -2464,8 +2464,12 @@
             raise KnitCorrupt(self._access,
                               'incorrect number of lines %s != %s'
                               ' for version {%s}'
+ ' (header = {%r}, last_line = {%r},'
+ ' record_contents = {%r}, rec = {%r})'
                               % (len(record_contents), int(rec[2]),
- version_id))
+ version_id,
+ header, last_line,
+ record_contents, rec),)
         if last_line != 'end %s\n' % rec[1]:
             raise KnitCorrupt(self._access,
                               'unexpected version end line %r, wanted %r'

[John Steele Scott (toojays)]

Sorry, launchpad mangled the whitespace on that diff. Posting as an attachment instead.

[John Steele Scott (toojays)]

Sorry, launchpad mangled the whitespace on that diff. Posting as an attachment instead.

[Andrew Straw (astraw)]

OK, I'm attaching the stderr output of my newly-patched 1.4rc2. Also, I did and svnadmin dump/load cycle on the repository and still I get the same error.

[Andrew Bennetts (spiv)]

Ah, it is similar to bug 3918, although slightly different.

The problematic part is:

symlink_target="t os, sys, datetime
" />

Note the newline inside the symlink target!

[Robert Collins (lifeless)]

On Fri, 2008-05-02 at 00:53 +0000, Andrew Bennetts wrote:
> Ah, it is similar to bug 3918, although slightly different.
>
> The problematic part is:
>
> symlink_target="t os, sys, datetime
> " />
>
> Note the newline inside the symlink target!

This suggests insufficient sanitisation/escaping by bzr-svn.

I don't think bzr supports such symlinks itself though.

-Rob
--
GPG key available at: <http://www.robertcollins.net/keys.txt>.

[John Steele Scott (toojays)]

Hmm, my dump output included an entry with:

symlink_target="###########################################################################
', '"

No idea what that's about. I wouldn't have thought our svn repo had any symlinks in it, but the revision which kills the import is way before my time, so I can't say for sure.

Adding bzr-svn project to this bug.

Can you suggest a workaround or quick fix for this bug? Or does the two-year-old bug 3918 need to be resolved first?

[Andrew Straw (astraw)]

OK, it looks like my repo had a commit in which a symlink was changed to a file in one step. When I tried to reproduce this on a new repository, I get

$ svn commit -m "change symlink to regular file"
svn: Commit failed (details follow):
svn: Entry '/home/astraw/src/symlink/hi-link' has unexpectedly changed special status

So, it looks like there was a bug in whatever version of svn allowed this commit to happen and the svn repo itself is broken.

I'll attempt to workaround the issue by re-building the svn repository without symlinks.

Thanks for the slueth work.

[Jelmer Vernooij (jelmer)]

Does anybody have a script for reproducing this?

[Jelmer Vernooij (jelmer)]

I just reproduced this.

bzr-svn will now check a bit better that special files are actually symlinks, so I'm closing the bzr-svn portion of this bug.

bzr doesn't support symlinks that point at filenames with newlines in them, so leaving the bzr task of this bug open. To reproduce:

bzr init foo
cd foo
ln -s "kamelot
epica" prologue
bzr add prologue
bzr ci -m break

[Jelmer Vernooij (jelmer)]

Subversion appears to support versioning symlinks to files with newlines in their name just fine, so we should be able to support them.

[Robert Collins (lifeless)]

On Mon, 2008-07-28 at 23:00 +0000, Jelmer Vernooij wrote:
> I just reproduced this.
>
> bzr-svn will now check a bit better that special files are actually
> symlinks, so I'm closing the bzr-svn portion of this bug.
>
> bzr doesn't support symlinks that point at filenames with newlines in
> them, so leaving the bzr task of this bug open. To reproduce:

bzr doesn't support *filenames* with newlines in them; I don't think we
have any plans to. Can you at least prevent bzr-svn from generating
corrupt inventory files?

-Rob
--
GPG key available at: <http://www.robertcollins.net/keys.txt>.

[Jelmer Vernooij (jelmer)]

I really think that's Bazaar's job since it's being used to generate those inventory files: it should be raising an appropriate exception when a file with a newline in it is being added to the inventory rather than error out this way.

Even if I make bzr-svn raise a more appropriate exception, Bazaar will still error out if you try to add a filename with a newline in it directly.

[Jelmer Vernooij (jelmer)]

Please note that no corrupt knit files are ever written, the exception occurs while generating the knit delta.

[Robert Collins (lifeless)]

On Tue, 2008-07-29 at 00:16 +0000, Jelmer Vernooij wrote:
> I really think that's Bazaar's job since it's being used to generate
> those inventory files: it should be raising an appropriate exception
> when a file with a newline in it is being added to the inventory rather
> than error out this way.
>
> Even if I make bzr-svn raise a more appropriate exception, Bazaar will
> still error out if you try t add a filename with a newline in it
> directly.

Ah, I thought bzr-svn generated xml directly for performance. If bzr is
generating the xml and failing then it is indeed a bzr task to change
further.

-Rob
--
GPG key available at: <http://www.robertcollins.net/keys.txt>.

[Martin Pool (mbp)]

I think this problem, to do with storing newlines in symlink targets, should be fixed in the 2a format of inventories. It can't be easily fixed in old formats. Therefore I think the bzr side of the bug is now fixed.