bzr fast-import from git fast-export yields incorrect branches

[Dustin Kirkland ]

Hello,

I've successfully moved all aspects of my upstream project from elsewhere into Launchpad, except for the source code repository, which is still git hosted on kernel.org.

I have tried to import using the following:

 $ mkdir /tmp/git
 $ cd /tmp/git
 $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/mhalcrow/ecryptfs-utils.git
 $ cd ecryptfs-utils
 $ git-fast-export --all > /tmp/out
 $ mkdir /tmp/bzr
 $ cd /tmp/bzr
 $ bzr init-repository .
 $ cat /tmp/out | bzr fast-import -
 $ mkdir /tmp/checkout
 $ cd /tmp/checkout
 $ bzr branch /tmp/bzr/master
 $ cd master

Note the incongruencies between /tmp/git/ecryptfs-utils and /tmp/checkout/master. For instance, the current git tree contains the file COPYING, while the bzr branch does not.

James Westby helped me with this quite a bit, though we're pretty much stuck at this point. Help? Thanks.

:-Dustin

[Jelmer Vernooij (jelmer)]

Hi Dustin,

On Wed, 2009-02-04 at 15:10 +0000, Dustin Kirkland wrote:
> I've successfully moved all aspects of my upstream project from
> elsewhere into Launchpad, except for the source code repository, which
> is still git hosted on kernel.org.
>
> I have tried to import using the following:
>
> $ mkdir /tmp/git
> $ cd /tmp/git
> $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/mhalcrow/ecryptfs-utils.git
> $ cd ecryptfs-utils
> $ git-fast-export --all > /tmp/out
> $ mkdir /tmp/bzr
> $ cd /tmp/bzr
> $ bzr init-repository .
> $ cat /tmp/out | bzr fast-import -
> $ mkdir /tmp/checkout
> $ cd /tmp/checkout
> $ bzr branch /tmp/bzr/master
> $ cd master
>
> Note the incongruencies between /tmp/git/ecryptfs-utils and
> /tmp/checkout/master. For instance, the current git tree contains the
> file COPYING, while the bzr branch does not.
>
> James Westby helped me with this quite a bit, though we're pretty much
> stuck at this point. Help? Thanks.
Doesn't really fix this particular issue in fastimport, but bzr-git
(seems) to support this branch without problems. I've pushed an example
branch to lp:~jelmer/+junk/ecryptfs-utils.

Cheers,

Jelmer

--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

[Dustin Kirkland  (kirkland)]

On Wed, Feb 4, 2009 at 4:48 PM, Jelmer Vernooij <email address hidden> wrote:
> Doesn't really fix this particular issue in fastimport, but bzr-git
> (seems) to support this branch without problems. I've pushed an example
> branch to lp:~jelmer/+junk/ecryptfs-utils.

Hi Jelmer-

I've been trying to use bzr-git as well, but haven't succeeded in
using it yet. I branch the source in my plugins directory and it
breaks with some ?foreign? issues. Can you give me explicit
instructions (or a pointer to explicit instructions) on how to install
the bzr-git plugin?

As far as I'm concerned, this is really a one-time operation. Once I
get the git source into Launchpad, and I'm satisfied that the history
is correctly imported and the current state of the master branch is
identical between the two, so long git ;-)

:-Dustin

[Jelmer Vernooij (jelmer)]

On Wed, 2009-02-04 at 16:01 +0000, Dustin Kirkland wrote:
> On Wed, Feb 4, 2009 at 4:48 PM, Jelmer Vernooij <email address hidden> wrote:
> > Doesn't really fix this particular issue in fastimport, but bzr-git
> > (seems) to support this branch without problems. I've pushed an example
> > branch to lp:~jelmer/+junk/ecryptfs-utils.

> I've been trying to use bzr-git as well, but haven't succeeded in
> using it yet. I branch the source in my plugins directory and it
> breaks with some ?foreign? issues. Can you give me explicit
> instructions (or a pointer to explicit instructions) on how to install
> the bzr-git plugin?
>
> As far as I'm concerned, this is really a one-time operation. Once I
> get the git source into Launchpad, and I'm satisfied that the history
> is correctly imported and the current state of the master branch is
> identical between the two, so long git ;-)
At the moment bzr-git requires python-dulwich (https://launchpad.net,
packaged in PPA at https://launchpad.net/~dulwich/) and the development
version of bzr in order to run. Of course, the resulting branch doesn't
require bzr-git in any way so should work with older versions of bzr as
well.

Cheers,

Jelmer

--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

[Dustin Kirkland  (kirkland)]

>> On Wed, Feb 4, 2009 at 4:48 PM, Jelmer Vernooij <email address hidden> wrote:
>> > Doesn't really fix this particular issue in fastimport, but bzr-git
>> > (seems) to support this branch without problems. I've pushed an example
>> > branch to lp:~jelmer/+junk/ecryptfs-utils.

$ bzr branch lp:~jelmer/+junk/ecryptfs-utils
bzr: ERROR: Unknown repository format: 'Bazaar
RepositoryFormatKnitPack6RichRoot (bzr 1.9)\n'

:-Dustin

[Jelmer Vernooij (jelmer)]

On Wed, 2009-02-04 at 16:30 +0000, Dustin Kirkland wrote:
> >> On Wed, Feb 4, 2009 at 4:48 PM, Jelmer Vernooij <email address hidden> wrote:
> >> > Doesn't really fix this particular issue in fastimport, but bzr-git
> >> > (seems) to support this branch without problems. I've pushed an example
> >> > branch to lp:~jelmer/+junk/ecryptfs-utils.
>
> $ bzr branch lp:~jelmer/+junk/ecryptfs-utils
> bzr: ERROR: Unknown repository format: 'Bazaar
> RepositoryFormatKnitPack6RichRoot (bzr 1.9)\n'
Please try again from lp:~jelmer/+junk/ecryptfs-utils-rrp, that should
be using a format that has been supported since bzr 1.0.

Cheers,

Jelmer

--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

[Ian Clatworthy (ian-clatworthy)]

Dustin,

Thanks for the bug report. I've grabbed the git repo, imported it, briefly browsed the various branches and the output from git fast-export ...

  (prepare input file using git fast-export)
  (look at input file and see that COPYING added to the refs/remotes/origin/devel branch)
  mkdir ecryptfs-utils.bzr
  cd ecryptfs-utils.bzr
  bzr init-repo .
  bzr fast-import ../ecryptfs-utils.fi
  cd master
  bzr update
  (COPYING not there)
  bzr log --line | less
  (hmm - no mention of merges from devel)
  cd ../devel
  bzr update
  (COPYING there)
  bzr log COPYING
  (added in rev 2 and no changes since)

If you're running bzr.dev (or 1.12rc I think), "bzr log --line" on a branch concisely shows the history of each branch and puts [merge] after the date each time a merge occurred. To see the merged revisions, "bzr log --line -n0".

Can you please try that and let me know how sensible or otherwise the branch histories look? If bzr-fastimport is doing the wrong thing wrt interpreting the git-fast-export stream, I suspect it will stand out pretty obviously to you. We can then go back to the input file and see where I'm making the wrong assumption, e.g. I might be using the wrong basis somewhere when a new branch starts?

[Ian Clatworthy (ian-clatworthy)]

This bug has me worried so I've tried really hard this weekend to track it down. I can confirm that the original git branches have a different number of revisions to the one generated by bzr-fastimport:

* devel: bzr=49, git=49
* escrow: bzr=197, git=231
* master: bzr=270, git=304
* nss: bzr=209, git=243
* ubuntu: bzr=226, git=270

I got those numbers by running

  bzr log --line -n0 xxx | wc -l

and

  git log --pretty=oneline --topo-order origin/xxx | wc -l

in the respective repositories. So most branches are missing 34 revisions and, looking deeper, those 34 revisions are only in the devel branch in the repo+branches generated by bzr-fastimport. *However*, I can't find anything in the data stream that suggests those revisions *ought* to be in the other branches. As best I can determine, bzr-fastimport is doing exactly the right thing and the bug is therefore either in git-fast-export or git-log or ???

Any ideas? Any further information you can give me here?

FWIW, I've merged a nice patch from Brian de Alwis so that bzr-fastimport can support resets. That has improved bzr-fastimport's tracking of heads which I thought might be related to this but it hasn't made any difference w.r.t the above. Grab rev 109 if you want to try out the latest code.

[Dustin Kirkland  (kirkland)]

Hi Ian-

Thanks so much for taking this bug so seriously ;-)

Jelmer actually provided me with a branch he was able to create using
bzr-git. I ran a number of checks over it to ensure that histories
were preserved, and that the head of the bzr branch and git tree were
precisely identical. I started using that and have been developing
off of it ever since.

To be honest, there isn't anything of real value in the other git
branches (besides the head), so I wasn't too worried about losing
them.

Still, the issue remains, as you were able to reproduce, that it's not
yet possible to do a simple git export and bzr import for this source
tree.

Unfortunately, I don't know anything about either git or bzr code
base, so I'm not of much help here. It could very well be a bug in
git's fast-export. I filed this bug here mainly to track it, and I
thought someone might be able to show me how I was misusing the tools
;-)

:-Dustin

[Ian Clatworthy (ian-clatworthy)]

IIUIC, the original repository has been effectively migrated and there's no obvious bug in fast-import here. We can re-open this is that's not the case.