Support leading zeroes in tree entry modes

Bug #620557 reported by Armin Ronacher
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Dulwich
Triaged
Medium
Unassigned

Bug Description

The following exception occurs when I check out the Flask repository from github (http://github.com/mitsuhiko/flask):

(env)mitsuhiko@nausicaa:~/Development/sourcesync/repos/flask$ hg pull git://github.com/mitsuhiko/flask.git
pulling from git://github.com/mitsuhiko/flask.git
importing Hg objects into Git
Counting objects: 3674, done.
Compressing objects: 100% (1588/1588), done.
Total 3674 (delta 2491), reused 3078 (delta 2025)
importing Git objects into Hg
** unknown exception encountered, details follow
** report bug details to http://mercurial.selenic.com/bts/
** or <email address hidden>
** Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) [GCC 4.2.1 (Apple Inc. build 5646)]
** Mercurial Distributed SCM (version 1.6.2)
** Extensions loaded: fetch, graphlog, mq, rebase, zeroconf, transplant, record, convert, attic, hgchangelog, bookmarks, hggit
Traceback (most recent call last):
  File "./env/bin/hg", line 27, in <module>
    mercurial.dispatch.run()
  File ".venv/mercurial/dispatch.py", line 16, in run
    sys.exit(dispatch(sys.argv[1:]))
  File ".venv/mercurial/dispatch.py", line 34, in dispatch
    return _runcatch(u, args)
  File ".venv/mercurial/dispatch.py", line 54, in _runcatch
    return _dispatch(ui, args)
  File ".venv/mercurial/dispatch.py", line 490, in _dispatch
    cmdpats, cmdoptions)
  File ".venv/mercurial/dispatch.py", line 351, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File ".venv/mercurial/dispatch.py", line 541, in _runcommand
    return checkargs()
  File ".venv/mercurial/dispatch.py", line 495, in checkargs
    return cmdfunc()
  File ".venv/mercurial/dispatch.py", line 488, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/mercurial/extensions.py", line 129, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/hgext/bookmarks.py", line 409, in pull
    result = oldpull(ui, repo, source, **opts)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/mercurial/extensions.py", line 129, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/hgext/rebase.py", line 530, in pullrebase
    orig(ui, repo, *args, **opts)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/mercurial/extensions.py", line 129, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/hgext/mq.py", line 2811, in mqcommand
    return orig(ui, repo, *args, **kwargs)
  File ".venv/mercurial/util.py", line 420, in check
    return func(*args, **kwargs)
  File ".venv/mercurial/commands.py", line 2803, in pull
    modheads = repo.pull(other, heads=revs, force=opts.get('force'))
  File ".venv/hggit/hgrepo.py", line 12, in pull
    git.fetch(remote.path, heads)
  File ".venv/hggit/git_handler.py", line 100, in fetch
    self.import_git_objects(remote_name, refs)
  File ".venv/hggit/git_handler.py", line 407, in import_git_objects
    self.import_git_commit(commit)
  File ".venv/hggit/git_handler.py", line 417, in import_git_commit
    files = self.get_files_changed(commit)
  File ".venv/hggit/git_handler.py", line 823, in get_files_changed
    for (oldfile, newfile), (oldmode, newmode), (oldsha, newsha) in changes:
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/object_store.py", line 142, in tree_changes
    ttree = self[tid]
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/object_store.py", line 107, in __getitem__
    return ShaFile.from_raw_string(type_num, uncomp)
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/objects.py", line 370, in from_raw_string
    obj.set_raw_string(string)
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/objects.py", line 257, in set_raw_string
    self.set_raw_chunks([text])
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/objects.py", line 261, in set_raw_chunks
    self._deserialize(chunks)
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/objects.py", line 837, in _deserialize
    self._entries = dict([(n, (m, s)) for n, m, s in parsed_entries])
  File ".venv/dulwich-0.6.2-py2.6-macosx-10.6-universal.egg/dulwich/objects.py", line 698, in parse_tree
    assert mode_text[0] != '0'
AssertionError

[traceback paths stripped for readability]

Using latest hggit for testing, mercurial 1.6 and dulwich's latest development version.

Jelmer Vernooij (jelmer)
summary: - Dulwich Fails checking out mitsuhiko/flask on github
+ Support leading zeroes in tree entry modes
Changed in dulwich:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Armin Ronacher (mitsuhiko) wrote :

Is there an ETA for this bug? I would have a look at it myself in case you are busy :)

Revision history for this message
ronny (ronny-pfannschmidt-deactivatedaccount) wrote :

i narrowed it down to compiled versus uncompiled

the c speedups don't "break",

Revision history for this message
Jelmer Vernooij (jelmer) wrote : Re: [Bug 620557] Re: Support leading zeroes in tree entry modes

On Mon, 2010-08-30 at 12:19 +0000, ronny wrote:
> i narrowed it down to compiled versus uncompiled
>
> the c speedups don't "break",
Ronny and I have discussed this a bit on IRC, but I thought I should
just mention it here as well so the bug reports has all data.

The C speedups don't have this assertion at the moment but they probably
should. The assertion is present because we would like to make sure that
when a Tree object is parsed by Dulwich that serializing it again will
result in the exact same text - with the same SHA1.

At the moment we don't handle leading zeroes so when reserializing a
tree a different text with a different SHA1 is generated.

Cheers,

Jelmer

Revision history for this message
Jelmer Vernooij (jelmer) wrote :

On Sat, 2010-08-28 at 20:52 +0000, Armin Ronacher wrote:
> Is there an ETA for this bug? I would have a look at it myself in case
> you are busy :)
It's somewhere on my todo list, though it's hard for me to give you an
ETA. The mitsuhiko/flask repository is the only one I've seen this issue
with so far.

Cheers,

Jelmer

Revision history for this message
Andrii Kurinnyi (andrew-marpasoft) wrote :

Just had the same issue with one of my private repos on GitHub
Chokes on the same mode as with mitsuhiko/flask: 040000
Removing assertion fixes the issue, though it is, probably, completely wrong solution.

Revision history for this message
Jon Frisby (jfrisby) wrote :

I've seen this with git://github.com/rails/rails.git, git://github.com/MrJoy/gollum.git, etc. Note that with the Rails repo it fails VERY far into the import process, so I'm thinking it may be that Git has changed to do this by default with new commits. If that's the case, then of course Dulwich would be pretty well useful for any new repos/commits until it's fixed...

Revision history for this message
Jon Frisby (jfrisby) wrote :

More info:

[11:27pm] AlexS: Found your bug.
[11:27pm] AlexS: It is not git
[11:27pm] AlexS: The format for file permissions has not changed
[11:27pm] AlexS: and git considers a leading zero to be a warning, and says as much in git fsck
[11:27pm] AlexS: GitHub fucked up grit
[11:28pm] AlexS: If you do an inline edit on github you get a leading 0.

Revision history for this message
Jon Frisby (jfrisby) wrote :

Oops, more info:

[11:30pm] AlexS: Anytime someone did that from March 11 through... I dunno, sometime later, maybe they still haven't fixed it, has generated a commit with a leading 0 in the permissions.
[11:30pm] jfrisby: Oh. Hunh.
[11:30pm] AlexS: And that is where they come from.
[11:30pm] AlexS: The octocat.
[11:30pm] AlexS: Much like babies.
[11:31pm] jfrisby: Is it just for that record in the tree, or is the entire tree object affected?
[11:31pm] AlexS: Due to the way git works, everything for ever and ever after that is affected.
[11:31pm] AlexS: You can't fix it without rewriting all history after that point.
[11:32pm] AlexS: Also I think it's the tree object.

Revision history for this message
Jelmer Vernooij (jelmer) wrote :

Thanks, that's very useful information.

So, it shouldn't be too hard to create a test case for this to test the parsing of a serialized tree object with leading zeroes in the permissions.

Jelmer Vernooij (jelmer)
Changed in dulwich:
milestone: none → 0.7.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.