zero-padded file modes break git->bzr imports

[Simone Margaritelli]

When i try to start a job to import my git repo into the bzr branch, the job fails and i get the following message in the logs :

2010-05-15 11:16:45 INFO Starting job.
2010-05-15 11:16:45 INFO Getting exising bzr branch from central store.
2010-05-15 11:16:45 INFO 35 bytes transferred |
2010-05-15 11:16:46 INFO Importing foreign branch.
2010-05-15 11:16:48 INFO Counting objects: 370
Counting objects: 377, done. 0
2010-05-15 11:16:49 INFO finding revisions to fetch:generating index 0/377
2010-05-15 11:16:49 INFO finding revisions to fetch 1/356
2010-05-15 11:16:49 INFO
Traceback (most recent call last):
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/scripts/code-import-worker.py", line 82, in <module>
    sys.exit(script.main())
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/scripts/code-import-worker.py", line 77, in main
    return import_worker.run()
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/lib/lp/codehosting/codeimport/worker.py", line 437, in run
    return self._doImport()
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/lib/lp/codehosting/codeimport/worker.py", line 575, in _doImport
    overwrite=True, **self.getExtraPullArgs())
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/branch.py", line 453, in pull
    stop_revision, overwrite=overwrite, graph=graph, limit=limit)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/branch.py", line 403, in _update_revisions
    determine_wants, self.source.mapping, limit=limit)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/fetch.py", line 535, in fetch_objects
    objects_iter, store, recorded_wants, pb, limit)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/fetch.py", line 420, in import_git_objects
    trees_cache)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/fetch.py", line 335, in import_git_commit
    allow_submodules=getattr(repo._format, "supports_tree_reference", False))
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/fetch.py", line 222, in import_git_tree
    base_tree = lookup_object(base_hexsha)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/fetch.py", line 371, in lookup_object
    return target_git_object_retriever[sha]
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/object_store.py", line 479, in __getitem__
    unusual_modes, expected_sha=sha)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/object_store.py", line 389, in _reconstruct_tree
    _check_expected_sha(expected_sha, tree)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/object_store.py", line 107, in _check_expected_sha
    expected_sha))
AssertionError: Invalid sha for <Tree 4a698785d734daf98f3aee3a3318f5b32d94de98>: 97f1ccb990d78f6b7a1c9aec69c367594c8fa56f
Import failed:
Traceback (most recent call last):
Failure: twisted.internet.error.ProcessTerminated: A process has ended with a probable error condition: process ended with exit code 1.

[Michael Hudson-Doyle (mwhudson)]

Which import is this?

[Simone Margaritelli (evilsocket)]

An import from the git repo on git://github.com/evilsocket/hybris.git .
Sometimes the import goes as it should, but most of the times it fails with this kind of error.

[Jelmer Vernooij (jelmer)]

Easiest way to reproduce this issue:

bzr branch git://github.com/evilsocket/hybris.git
cd hybris.git
rm -rf .bzr/repository/git
bzr git-objects

[Jelmer Vernooij (jelmer)]

This is caused by zero-padded file modes in the git repository. To see the warnings from git:

$ git clone git://github.com/evilsocket/hybris.git
$ cd hybris.git
$ git fsck --full
warning in tree 0a25d5f22d8677b38b14e3b3f8d6ad815782e0f8: contains zero-padded file modes
warning in tree 438c91c596ec31f0a4d42ff7d361db8c505ebd2a: contains zero-padded file modes
warning in tree 662f9059b67ba7f95565b694286ff336362890d1: contains zero-padded file modes
warning in tree a40b6cb8252533e3bf3fa9d19366db5bbb3b2cd7: contains zero-padded file modes
warning in tree c170762f3e4be04f5202ce37e4993ce334e361d8: contains zero-padded file modes
warning in tree ce6c5fd1b863da641cabaaaad4b1c16014126288: contains zero-padded file modes
warning in tree ea01d2b07b93a61f1cd9c56a9c0d8103d25914ae: contains zero-padded file modes

[Jelmer Vernooij (jelmer)]

Apparently this is caused by a bug in github's "gist" feature, which causes it to write out incorrect data. c git silently ignores this error but dulwich and jgit both error out.

[Robert Collins (lifeless)]

@jelmer, what is needed in LP here?

[Jelmer Vernooij (jelmer)]

On Sat, 2011-04-09 at 06:12 +0000, Robert Collins wrote:
> @jelmer, what is needed in LP here?
lp:bzr-git needs to be merged into lp:~launchpad-pqm/bzr-git/devel and
sourcedeps.conf needs to be updated to use a new bzr-git revno.

This bug is mainly here since it affects launchpad code imports (that's
why it was originally reported) and to give users an ability to track
its progress in Launchpad.

Cheers,

Jelmer

[Robert Collins (lifeless)]

@jelmer is that merge doable now? Would you like to put it forward if it is?

[xaav (xaav)]

This is now affecting my project; linking question.

[Jelmer Vernooij (jelmer)]

@lifeless The merge for the launchpad imports can't happen until the bug is fixed in bzr-git.

[xaav (xaav)]

Since this is really annoying, and since the fix really isn't that hard, I'm going to try to fix this.

Line: http://bazaar.launchpad.net/~bzr-git/bzr-git/trunk/view/head:/object_store.py#L146

[xaav (xaav)]

Sorry, the fix appears to be more complex than I thought it was going to be, and I'm not equipped to deal with that.

[Robert Collins (lifeless)]

Marking low for LP, as it was suffering starvation from the low priority of the dependent tasks.