Support leading zeroes in tree entry modes

[Armin Ronacher]

The following exception occurs when I check out the Flask repository from github (http://github.com/mitsuhiko/flask):

(env)mitsuhiko@nausicaa:~/Development/sourcesync/repos/flask$ hg pull git://github.com/mitsuhiko/flask.git
pulling from git://github.com/mitsuhiko/flask.git
importing Hg objects into Git
Counting objects: 3674, done.
Compressing objects: 100% (1588/1588), done.
Total 3674 (delta 2491), reused 3078 (delta 2025)
importing Git objects into Hg
** unknown exception encountered, details follow
** report bug details to http://mercurial.selenic.com/bts/
** or <email address hidden>
** Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) [GCC 4.2.1 (Apple Inc. build 5646)]
** Mercurial Distributed SCM (version 1.6.2)
** Extensions loaded: fetch, graphlog, mq, rebase, zeroconf, transplant, record, convert, attic, hgchangelog, bookmarks, hggit
Traceback (most recent call last):
  File "./env/bin/hg", line 27, in <module>
    mercurial.dispatch.run()
  File ".venv/mercurial/dispatch.py", line 16, in run
    sys.exit(dispatch(sys.argv[1:]))
  File ".venv/mercurial/dispatch.py", line 34, in dispatch
    return _runcatch(u, args)
  File ".venv/mercurial/dispatch.py", line 54, in _runcatch
    return _dispatch(ui, args)
  File ".venv/mercurial/dispatch.py", line 490, in _dispatch
    cmdpats, cmdoptions)
  File ".venv/mercurial/dispatch.py", line 351, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File ".venv/mercurial/dispatch.py", line 541, in _runcommand
    return checkargs()
  File ".venv/mercurial/dispatch.py", line 495, in checkargs
    return cmdfunc()
  File ".venv/mercurial/dispatch.py", line 488, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/mercurial/extensions.py", line 129, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/hgext/bookmarks.py", line 409, in pull
    result = oldpull(ui, repo, source, **opts)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/mercurial/extensions.py", line 129, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/hgext/rebase.py", line 530, in pullrebase
    orig(ui, repo, *args, **opts)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/mercurial/extensions.py", line 129, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/hgext/mq.py", line 2811, in mqcommand
    return orig(ui, repo, *args, **kwargs)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/mercurial/commands.py", line 2803, in pull
    modheads = repo.pull(other, heads=revs, force=opts.get('force'))
  File ".venv/hggit/hgrepo.py", line 12, in pull
    git.fetch(remote.path, heads)
  File ".venv/hggit/git_handler.py", line 100, in fetch
    self.import_git_objects(remote_name, refs)
  File ".venv/hggit/git_handler.py", line 407, in import_git_objects
    self.import_git_commit(commit)
  File ".venv/hggit/git_handler.py", line 417, in import_git_commit
    files = self.get_files_changed(commit)
  File ".venv/hggit/git_handler.py", line 823, in get_files_changed
    for (oldfile, newfile), (oldmode, newmode), (oldsha, newsha) in changes:
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/object_store.py", line 142, in tree_changes
    ttree = self[tid]
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/object_store.py", line 107, in __getitem__
    return ShaFile.from_raw_string(type_num, uncomp)
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/objects.py", line 370, in from_raw_string
    obj.set_raw_string(string)
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/objects.py", line 257, in set_raw_string
    self.set_raw_chunks([text])
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/objects.py", line 261, in set_raw_chunks
    self._deserialize(chunks)
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/objects.py", line 837, in _deserialize
    self._entries = dict([(n, (m, s)) for n, m, s in parsed_entries])
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/objects.py", line 698, in parse_tree
    assert mode_text[0] != '0'
AssertionError

[traceback paths stripped for readability]

Using latest hggit for testing, mercurial 1.6 and dulwich's latest development version.

[Armin Ronacher (mitsuhiko)]

Is there an ETA for this bug? I would have a look at it myself in case you are busy :)

[ronny (ronny-pfannschmidt-deactivatedaccount)]

i narrowed it down to compiled versus uncompiled

the c speedups don't "break",

[Jelmer Vernooij (jelmer)]

On Mon, 2010-08-30 at 12:19 +0000, ronny wrote:
> i narrowed it down to compiled versus uncompiled
>
> the c speedups don't "break",
Ronny and I have discussed this a bit on IRC, but I thought I should
just mention it here as well so the bug reports has all data.

The C speedups don't have this assertion at the moment but they probably
should. The assertion is present because we would like to make sure that
when a Tree object is parsed by Dulwich that serializing it again will
result in the exact same text - with the same SHA1.

At the moment we don't handle leading zeroes so when reserializing a
tree a different text with a different SHA1 is generated.

Cheers,

Jelmer

[Jelmer Vernooij (jelmer)]

On Sat, 2010-08-28 at 20:52 +0000, Armin Ronacher wrote:
> Is there an ETA for this bug? I would have a look at it myself in case
> you are busy :)
It's somewhere on my todo list, though it's hard for me to give you an
ETA. The mitsuhiko/flask repository is the only one I've seen this issue
with so far.

Cheers,

Jelmer

[Andrii Kurinnyi (andrew-marpasoft)]

Just had the same issue with one of my private repos on GitHub
Chokes on the same mode as with mitsuhiko/flask: 040000
Removing assertion fixes the issue, though it is, probably, completely wrong solution.

[Jon Frisby (jfrisby)]

I've seen this with git://github.com/rails/rails.git, git://github.com/MrJoy/gollum.git, etc. Note that with the Rails repo it fails VERY far into the import process, so I'm thinking it may be that Git has changed to do this by default with new commits. If that's the case, then of course Dulwich would be pretty well useful for any new repos/commits until it's fixed...

[Jon Frisby (jfrisby)]

More info:

[11:27pm] AlexS: Found your bug.
[11:27pm] AlexS: It is not git
[11:27pm] AlexS: The format for file permissions has not changed
[11:27pm] AlexS: and git considers a leading zero to be a warning, and says as much in git fsck
[11:27pm] AlexS: GitHub fucked up grit
[11:28pm] AlexS: If you do an inline edit on github you get a leading 0.

[Jon Frisby (jfrisby)]

Oops, more info:

[11:30pm] AlexS: Anytime someone did that from March 11 through... I dunno, sometime later, maybe they still haven't fixed it, has generated a commit with a leading 0 in the permissions.
[11:30pm] jfrisby: Oh. Hunh.
[11:30pm] AlexS: And that is where they come from.
[11:30pm] AlexS: The octocat.
[11:30pm] AlexS: Much like babies.
[11:31pm] jfrisby: Is it just for that record in the tree, or is the entire tree object affected?
[11:31pm] AlexS: Due to the way git works, everything for ever and ever after that is affected.
[11:31pm] AlexS: You can't fix it without rewriting all history after that point.
[11:32pm] AlexS: Also I think it's the tree object.

[Jelmer Vernooij (jelmer)]

Thanks, that's very useful information.

So, it shouldn't be too hard to create a test case for this to test the parsing of a serialized tree object with leading zeroes in the permissions.