set of rename operations breaks dpush

[Alexander Regueiro]

I am trying to push my ircdotnet project (lp:ircdotnet/0.4 branch) to GitHub using dpush, but am running into what looks like a database error with Git.

The following is my command line:

> bzr dpush git+ssh://<email address hidden>/alexreg/ircdotnet.git
error: unable to find fb1c6f6b77779fe32a31c04494b385bc53aac21f
fatal: object of unexpected type
bzr: ERROR: unpack index-pack abnormal exit

I've attached the relevant parts of my .bzr.log file recorded during a single failed dpush.

[Jelmer Vernooij (jelmer)]

The issue seems to be r47, directory "src":

------------------------------------------------------------
revno: 47
fixes bug(s): https://launchpad.net/bugs/809359 https://launchpad.net/bugs/808619
committer: Alex Regueiro <email address hidden>
branch nick: 0.4
timestamp: Wed 2011-07-13 00:49:44 +0100
message:
  Restructured working tree so that samples/ and tests/ are root dirs.
  Fixed bug where user information (real name/user name/host name) was sometimes incomplete when user joined channel.
  ModesChanged and TopicChanged events on IrcChannel now specify IrcUser that made change.
added:
  tests/
  tests/IrcDotNet-Tests.sln
  tests/IrcDotNet/App.xaml
  tests/IrcDotNet/App.xaml.cs
  tests/IrcDotNet/CompleteIrcClientOrderedTest.cs
  tests/IrcDotNet/IrcDotNet-Silverlight.Tests.csproj
  tests/IrcDotNet/OrderedTest.cs
  tests/IrcDotNet/Properties/OutOfBrowserSettings.xml
renamed:
  src/IrcDotNet.Tests/ => tests/IrcDotNet/
  src/IrcDotNet.Tests/README.md => tests/README.md
  src/IrcDotNet.vsmdi => tests/IrcDotNet.vsmdi
  src/Local.testsettings => tests/Local.testsettings
  src/Samples/ => samples/
  src/TraceAndTestImpact.testsettings => tests/TraceAndTestImpact.testsettings
modified:
  src/IrcDotNet.sln
  src/IrcDotNet/IrcChannel.cs
  src/IrcDotNet/IrcClient.cs
  src/IrcDotNet/IrcClientMessageProcessing.cs
  src/IrcDotNet/IrcDotNet-Silverlight.csproj
  src/IrcDotNet/IrcUtilities.cs
  tests/IrcDotNet/CompleteIrcClient.orderedtest
  tests/IrcDotNet/IrcClientTestSet.cs
  tests/IrcDotNet/IrcDotNet.Tests.csproj
  tests/IrcDotNet/Properties/AppManifest.xml
  tests/IrcDotNet/Properties/AssemblyInfo.cs
  tests/README.md
  tests/IrcDotNet.vsmdi
HPSS calls: 61 (43 vfs) SmartSSHClientMedium(bzr+ssh://<email address hidden>/)

the _revision_to_objects method doesn't generate a proper tree object for the src directory here, but it does point the root tree at it for the src/ branch.

[Alexander Regueiro (alexreg)]

I've verified that the problem is definitely caused by the merges into tests/. Further information upcoming.

[Jelmer Vernooij (jelmer)]

really useful here would be having a script which reproduces this issue from scratch in two revisions (which should be possible)

[Alexander Regueiro (alexreg)]

I've now managed to reproduce the issue from scratch, by taking the state of 0.4 r46, and manually making the appropriate changes (just renames and modifications) for r46 -> r47. Note that the renames alone do not cause the issue; only the combination of the two, which may suggest the interplay/ordering of the two sets of operations is the root cause of the issue.

[Alexander Regueiro (alexreg)]

Actually, what I said last time isn't quite true. I get the following message and error when manually applying the renames in r47 on a new branch containing r46 contents.

src/IrcDotNet.Tests => tests/IrcDotNet
bzr: ERROR: File id {readme.md-20110918173546-6gyz80j7trj2sieh-80} already exists in inventory as InventoryFile('readme.md-20110918173546-6gyz80j7trj2sieh-80', u'README.md', parent_id='ircdotnet.tests-20110918173546-6gyz80j7trj2sieh-18', sha1=None, len=None, revision=None)
src/IrcDotNet.vsmdi => tests/IrcDotNet.vsmdi
src/Local.testsettings => tests/Local.testsettings
src/Samples => samples
src/TraceAndTestImpact.testsettings => tests/TraceAndTestImpact.testsettings
Committing to: C:/Users/Alex/Documents/Visual Studio 2010/Projects/IRC.NET/dump/foo-bzr/
renamed src/Samples => samples
added tests
renamed src/IrcDotNet.Tests => tests/IrcDotNet
renamed src/IrcDotNet.vsmdi => tests/IrcDotNet.vsmdi
renamed src/Local.testsettings => tests/Local.testsettings
renamed src/TraceAndTestImpact.testsettings => tests/TraceAndTestImpact.testsettings

Most notable is the 2nd line which suggests src/IrcDotNet.Tests/README.md => tests/README.md is not working nicely in bzr-git. Perhaps we are getting closer here?

[Alexander Regueiro (alexreg)]

Okay, so I've now created a minimal example of a branch with 3 directories and 1 final. Hopefully this will help debug more easily now.

[Alexander Regueiro (alexreg)]

Here's the PowerShell 2.0 script (annotated) that reproduces the issue on a clean branch. It is designed to be run from an empty directory along-side the 0.4 branch (lp:ircdotnet/0.4), since it pulls certain stuff from there.

[Jelmer Vernooij (jelmer)]

That still uses your 0.4 branch. I'm really looking for a script that just creates dummy files ("echo foo > bar"), since I don't really want to ship ircdotnet with the bzr-git testsuite :-)

[Alexander Regueiro (alexreg)]

Okay, more good news. Attached is a PowerShell script that reproduces the issue from scratch. Just run the script in an arbitrary empty directory, and you should see the problem appear on the dpush! The script (perhaps converted to bash-script) would be suitable as a unit test I'd think.

[Jelmer Vernooij (jelmer)]

Thanks! I've added a unit test that reproduce this issue in lp:~jelmer/bzr-git/rename-issue.

[Alexander Regueiro (alexreg)]

A bit of debugging with the actual bzr-git code (latest version) has revealed that the problem specifically stems from the fact that git does not version empty directories, and this create de-synchronisation with the bzr branch. Specifically, I believe that bzr-git creates a key (hex-sha?) entry for the empty directory in revision 1 of the test branch, but does not actually exist in the git branch. However, when trying to dpush revision 2, bzr-git tries to perform the file move/rename into a directory that the git branch does not yet have. Hence, I propose a fix should check for the existence of all parent directories of a file-move target before performing the move. Let me know if this makes sense to you.

[Jelmer Vernooij (jelmer)]

revision_to_objects is more complicated than it needs to be. It needs
to first gather the newly generated blob objects and then, based on
that, the new tree objects similar to the way this is being done in
dulwich' commit_index.

One of the challenges here will be retrieving git Tree objects from the
base revision, so they can be modified.

[ChriS (christophe-troestler)]

I got a problem that maybe could be related to the fact that git is not tracking directories. A collaborator added 2 files in a new directory. When I did a merge [2] I got the files but they were then somehow "unreferenced" and so disappeared from the repository — my colleague needed to re-add them [3]. No deletion can be seen in the commit log.

[1] https://github.com/agarwal/ocamlweb/commit/e0f262034dbd2a1c194d65166df2887ca06a2043
[2] https://github.com/agarwal/ocamlweb/commit/cb436a86c21023d52a5b10e10e5a828beb35bbfb
[3] https://github.com/agarwal/ocamlweb/commit/8591b20395b0a71f59c1c439ac7863221e7a9aa3

[Alexander Regueiro (alexreg)]

Any progress on this bug?