bzr-fastexport crashed in git-bzr -- broken pipe

[Matěj Cepl]

Using git-bzr from http://github.com/pieter/git-bzr/ I get the attached crash after this output:

[matej@viklef git]$ git bzr add pardus-upstream ../pardus-cpan-zim/
Bazaar branch pardus-upstream added. You can fetch it with `git bzr fetch pardus-upstream`
[matej@viklef git]$ git bzr fetch pardus-upstream
There doesn't seem to be an existing refmap. Doing an initial import
*** BzrFastExport: Exported 0 commits; forcing checkpoint
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/about.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/bugs.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/development
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/development.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/faq.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/features.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/install
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/install.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/install/unix.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/install/windows.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/usage
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/usage.txt
fatal: Path share/zim/doc/Zim/install/unix.txt not in branch
fast-import: dumping crash report to .git/fast_import_crash_4557
Traceback (most recent call last):
  File "/home/matej/.bazaar/plugins/fastimport/exporters/bzr-fast-export.py", line 376, in <module>
    sys.exit(exporter.run())
  File "/home/matej/.bazaar/plugins/fastimport/exporters/bzr-fast-export.py", line 79, in run
    self.emit_commit(revid, options.git_branch)
  File "/home/matej/.bazaar/plugins/fastimport/exporters/bzr-fast-export.py", line 239, in emit_commit
    sys.stdout.write('data %d\n%s\n' % (len(text), text))
IOError: [Errno 32] Broken pipe

fast_import_crash_4557 has been

[Pieter (frim)]

Looks like a bug in the recursive rename functionality I added. The relevant part is:

  R share/zim/doc/Zim/install share/zim/doc/Zim/Install
  R share/zim/doc/Zim/install.txt share/zim/doc/Zim/Install.txt
* R share/zim/doc/Zim/install/unix.txt share/zim/doc/Zim/Install/Unix.txt

* R share/zim/doc/Zim/install/unix.txt share/zim/doc/Zim/Install/Unix.txt

should have been:

* R share/zim/doc/Zim/Install/unix.txt share/zim/doc/Zim/Install/Unix.txt

I'm not sure if the rename stuff is only broken on case-insensitive equal strings, or in general.

I'll try to take a look at it later on.

[Pieter (frim)]

The problem was the use of a dictionary rather than a array, which meant the renames would be done in a random order.

I pushed out a fix on https://code.launchpad.net/~frim/bzr-fastimport/export-fixes . I'll do a merge request to the bzr-fastimport maintainer. See also the attached patch.

[Matěj Cepl (mcepl)]

Not much better:

[matej@viklef zim]$ rm -rf git
[matej@viklef zim]$ mkdir git
[matej@viklef zim]$ cd git
[matej@viklef git]$ git init
Initialized empty Git repository in /home/matej/redhat/kompiliste/zim/git/.git/
[matej@viklef git]$ git bzr add pardus-upstream ../pardus-cpan-zim/
Bazaar branch pardus-upstream added. You can fetch it with `git bzr fetch pardus-upstream`
[matej@viklef git]$ git bzr fetch pardus-upstream
[matej@viklef git]$ git bzr fetch pardus-upstream
There doesn't seem to be an existing refmap. Doing an initial import
*** BzrFastExport: Exported 0 commits; forcing checkpoint
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/about.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/bugs.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/development
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/development.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/development.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/faq.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/features.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/install
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/install.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/install.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/install/unix.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/Install/unix.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/install/windows.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/Install/windows.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/usage
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/usage.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/usage.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/Development.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/Install.txt
*** BzrFastExport: Fixing recursive rename for share/zim/doc/Zim/Usage.txt
fatal: Path share/zim/doc/Zim/About.txt not in branch
fast-import: dumping crash report to .git/fast_import_crash_16336
Traceback (most recent call last):
  File "/home/matej/.bazaar/plugins/fastimport/exporters/bzr-fast-export.py", line 376, in <module>
    sys.exit(exporter.run())
  File "/home/matej/.bazaar/plugins/fastimport/exporters/bzr-fast-export.py", line 79, in run
    self.emit_commit(revid, options.git_branch)
  File "/home/matej/.bazaar/plugins/fastimport/exporters/bzr-fast-export.py", line 239, in emit_commit
    sys.stdout.write('data %d\n%s\n' % (len(text), text))
IOError: [Errno 32] Broken pipe
[matej@viklef git]$

[Matěj Cepl (mcepl)]

Just to show that I have applied patch correctly

[Pieter (frim)]

Ah yes, apparently a recent other change broke this:

http://bazaar.launchpad.net/~bzr/bzr-fastimport/fastimport.dev/revision/87

I'm not sure why that change was made, but that breaks your issue. If you revert that commit, it does work. You should probably contact Ian or Jonas for that.

[Matěj Cepl (mcepl)]

not yet

[Pieter (frim)]

Are you sure you didn't lose my patch in the process?

[Matěj Cepl (mcepl)]

Concerning https://bugs.launchpad.net/bzr-fastimport/+bug/287785/comments/7:

no, I got in my patches -- when I deapplied revision 87 and applied your patch http://launchpadlibrarian.net/18824084/0001-bzr-fast-export-Use-an-array-for-rename-bookkeeping.patch then git-bzr works as it should.

[Matěj Cepl (mcepl)]

So, now the question is how to get your patch on the top of revision 87 ... ;-)

[Pieter (frim)]

patch 87 is wrong. rather than moving the deletions to the top, the deletions should get the same recursive rename treatment. I might be able to patch it up.

[Pieter (frim)]

I merged the bzr-fastimport.dev and committed a right fix for 87 in there.

[Matěj Cepl (mcepl)]

Yeah, works great! Thanks

[Ian Clatworthy (ian-clatworthy)]

Merged to trunk now - rev #88.