new marks file format not supported

[Ryan Niebur]

Hi!

The following script results in a traceback from bzr-fastimport. This is a minimal test case of what the git-bzr code is doing.

Thanks,
Ryan

rm -fr bzr-temp git-temp
mkdir bzr-temp
cd bzr-temp
bzr init
touch a
bzr add a
bzr commit -m "yep"
cd ..
mkdir git-temp
cd git-temp
git init
bzr fast-export --export-marks=.git/bzr-upstream --git-branch=bzr/upstream ../bzr-temp > .git/temp1
git fast-import --export-marks=.git/git-upstream < .git/temp1
git checkout -b local_branch bzr/upstream
touch b
git add b
git commit -m "yepppppppppp"
git fast-export --import-marks=.git/git-upstream --export-marks=.git/git-upstream HEAD > .git/temp2
cd ../bzr-temp
bzr fast-import --import-marks=../git-temp/.git/bzr-upstream --export-marks=../git-temp/.git/bzr-upstream ../git-temp/.git/temp2

[Gonéri Le Bouder (goneri)]

rev 218

[Ian Clatworthy (ian-clatworthy)]

I had to back out this fix as it was making fast-export produce bad output. I'll try changing how marks are input/out as I think that's actually the core problem.

[Ian Clatworthy (ian-clatworthy)]

Please try the ~ian-clatworthy/bzr-fastimport/new-marks-format branch and see if that fixes the problem.

[Ryan Niebur (ryan52)]

hm, I installed from that branch, but now bzr gives me this error:
Unable to load plugin 'fastimport' from '/usr/lib/python2.5/site-packages/bzrlib/plugins'

[Ryan Niebur (ryan52)]

fwiw, I get this error:

bash jade master# git-temp $ python /usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/__init__.py
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/__init__.py", line 123, in <module>
    class cmd_fast_import(Command):
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/__init__.py", line 298, in cmd_fast_import
    value_switches=False, title='Repository format'),
TypeError: __init__() got an unexpected keyword argument 'lazy_registry'
bash jade master# git-temp $

[Ian Clatworthy (ian-clatworthy)]

Ryan Niebur wrote:
> fwiw, I get this error:
>
> bash jade master# git-temp $ python /usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/__init__.py
> Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/__init__.py", line 123, in <module>
> class cmd_fast_import(Command):
> File "/usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/__init__.py", line 298, in cmd_fast_import
> value_switches=False, title='Repository format'),
> TypeError: __init__() got an unexpected keyword argument 'lazy_registry'
> bash jade master# git-temp $
>

What version of Bazaar are you running?

Ian C.

[Ryan Niebur (ryan52)]

On Sun, Aug 23, 2009 at 10:48:12PM -0000, Ian Clatworthy wrote:
> Ryan Niebur wrote:
> > fwiw, I get this error:
> >
> > bash jade master# git-temp $ python /usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/__init__.py
> > Traceback (most recent call last):
> > File "/usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/__init__.py", line 123, in <module>
> > class cmd_fast_import(Command):
> > File "/usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/__init__.py", line 298, in cmd_fast_import
> > value_switches=False, title='Repository format'),
> > TypeError: __init__() got an unexpected keyword argument 'lazy_registry'
> > bash jade master# git-temp $
> >
>
> What version of Bazaar are you running?
>

1.5. that's old...I'll upgrade and retry in a bit.

--
_________________________
Ryan Niebur
<email address hidden>

[Ryan Niebur (ryan52)]

okay, upgrading bzr made the plugin load. now I have this problem:

bash jade bzr-temp $ bzr fast-import --import-marks=../git-temp/.git/bzr-upstream --export-marks=../git-temp/.git/bzr-upstream ../git-temp/.git/temp2
16:49:35 Collecting statistics ...
16:49:35 Starting import ...
bzr: ERROR: exceptions.KeyError: 0

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 835, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 1030, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 647, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/__init__.py", line 339, in run
    params, verbose)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/__init__.py", line 108, in _run
    return proc.process(p.iter_commands)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/processor.py", line 95, in process
    self._process(command_iter)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/processors/generic_processor.py", line 275, in _process
    processor.ImportProcessor._process(self, command_iter)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/processor.py", line 109, in _process
    self.pre_process()
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport/processors/generic_processor.py", line 134, in pre_process
    self.cache_mgr.revision_ids = mark_info[0]
KeyError: 0

bzr 1.17 on python 2.5.4 (linux2)
arguments: ['/usr/bin/bzr', 'fast-import', '--import-marks=../git-temp/.git/bzr-upstream', '--export-marks=../git-temp/.git/bzr-upstream', '../git-temp/.git/temp2']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.17]
  fastimport /usr/lib/python2.5/site-packages/bzrlib/plugins/fastimport [0.9dev]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [1.17]
  netrc_credential_store /usr/lib/python2.5/site-packages/bzrlib/plugins/netrc_credential_store [1.17]
  rebase /usr/lib/python2.5/site-packages/bzrlib/plugins/rebase [0.3]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.
bash jade bzr-temp $

[Gonéri Le Bouder (goneri)]

Ian,

Do you remember what was the problem with my patch?

[Matthew Brett (matthew-brett)]

With bzr 2.0.2 and current trunk (rev 268(, Ryan's script (and the several git-bzr scripts, which use the same idea) still crash with:

ABORT: exception occurred processing commit :3
bzr: ERROR: exceptions.KeyError: ':1'

Traceback (most recent call last):
  File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 842, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 1037, in run_bzr
    ret = run(*run_argv)
  File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 654, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/Users/mb312/.bazaar/plugins/fastimport/__init__.py", line 384, in run
    params, verbose, user_map=user_map)
  File "/Users/mb312/.bazaar/plugins/fastimport/__init__.py", line 111, in _run
    return proc.process(p.iter_commands)
  File "/Users/mb312/.bazaar/plugins/fastimport/processor.py", line 95, in process
    self._process(command_iter)
  File "/Users/mb312/.bazaar/plugins/fastimport/processors/generic_processor.py", line 280, in _process
    processor.ImportProcessor._process(self, command_iter)
  File "/Users/mb312/.bazaar/plugins/fastimport/processor.py", line 117, in _process
    handler(self, cmd)
  File "/Users/mb312/.bazaar/plugins/fastimport/processors/generic_processor.py", line 490, in commit_handler
    handler.process()
  File "/Users/mb312/.bazaar/plugins/fastimport/processor.py", line 200, in process
    self.pre_process_files()
  File "/Users/mb312/.bazaar/plugins/fastimport/bzr_commit_handler.py", line 582, in pre_process_files
    super(InventoryDeltaCommitHandler, self).pre_process_files()
  File "/Users/mb312/.bazaar/plugins/fastimport/bzr_commit_handler.py", line 84, in pre_process_files
    for p in parents]
KeyError: ':1'

bzr 2.0.2 on python 2.6.1 (Darwin-10.2.0-i386-64bit)
arguments: ['/usr/local/bin/bzr', 'fast-import', '--import-marks=../git-temp/.git/bzr-upstream', '--export-marks=../git-temp/.git/bzr-upstream', '../git-temp/.git/temp2']

I can code in python - is there anything I can do to help fix this problem?

Thanks a lot,

Matthew

[SirVer (sirver)]

I have the same problems at the same setup. I take the freedom to open this bug again, because it seems to be not fixed.

[SirVer (sirver)]

I had a little longer look into this and I think the problem is with some colomns (:). I am not aware of how the fast-import file format should look like, but the attached patch allows me to push back to bzr from git exactly once. Note though that pulling further changes from bzr doesn't work:

bzr$ bzr commit -m "Latest"
bzr$ cd ../git
git$ git bzr fetch origin # this works
hack hack
git$ git -am "Some commit" && git bzr push origin # This works with the attached patch
git$ cd ../bzr
hack hack
bzr$ bzr commit -m "More bzr changes"
bzr$ cd ../git
git$ git bzr fetch origin # This fails. But removing the bzr marks file (which makes bzr reexport everything) and repeating suceeds.

I am a bit lost here and I would need someone with more insight for coaching.

[SirVer (sirver)]

Damn.. always forgetting something :(. It seems as if git is exporting it's refids always with a colum ":1", ":2" while bzr expects them (and exports them) without "1","2", "3". Hope this helps someone helping us.

[Gabriel Filion (gabriel-filion)]

just a small hint: could this be caused by the fact that git actually doesn't store empty files in commits (thus not finding anything that points to the example commits made in the first example of this bug report -- touch a, bzr add a, bzr commit)?

[SirVer (sirver)]

Gabriel, while you are right that this is a bug in the report (because git doesn't track empty files), the error goes deeper than that. My bug description (#12) is still persistent with or without my patch.

[Matthew Brett (matthew-brett)]

Hi,

On Wed, Feb 10, 2010 at 8:59 AM, Gabriel Filion
<email address hidden> wrote:
> just a small hint: could this be caused by the fact that git actually
> doesn't store empty files in commits (thus not finding anything that
> points to the example commits made in the first example of this bug
> report -- touch a, bzr add a, bzr commit)?

Do you mean, that bzr fastimport cannot deal with empty git commits?
Is this a bug or a feature?

Thanks a lot,

Matthew

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matthew Brett wrote:
> Hi,
>
> On Wed, Feb 10, 2010 at 8:59 AM, Gabriel Filion
> <email address hidden> wrote:
>> just a small hint: could this be caused by the fact that git actually
>> doesn't store empty files in commits (thus not finding anything that
>> points to the example commits made in the first example of this bug
>> report -- touch a, bzr add a, bzr commit)?
>
> Do you mean, that bzr fastimport cannot deal with empty git commits?
> Is this a bug or a feature?
>
> Thanks a lot,
>
> Matthew
>

So if you add an empty file, which cannot be tracked in git, and commit
it, I would guess that you would get a new Commit object, but not a new
Tree object. I don't know if git would refuse a new Commit without a new
Tree, but I would think it would support importing that. (After all, you
could always be reverting back to a previous state, which is identical
as well.)

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkty/VgACgkQJdeBCYSNAAOXAACghYSy+3Rr2B/baK/vTWfQoAH0
/1UAn12W+JUz3lBvhnRnE2HzjnjcE5JH
=szdz
-----END PGP SIGNATURE-----

[Miklos Vajna (vmiklos)]

On Wed, Feb 10, 2010 at 05:46:59PM -0000, SirVer <email address hidden> wrote:
> Gabriel, while you are right that this is a bug in the report (because
> git doesn't track empty files), the error goes deeper than that. My bug
> description (#12) is still persistent with or without my patch.

Exactly, this has nothing with empty files. Anyway git *does* track
empty files, it just doesn't track empty directories.

Regarding "how should it look like", it's up to the given
vcs-fast-import/export implementation, it just has to be consistent.
FWIW both git and darcs implementations use: one ":foo bar" / line,
where foo is an int (mark number) and bar is the (persistent) id of the
commit/patch.

[Xavier (Open ERP) (xmo-deactivatedaccount)]

Broken as well in here, using kfish's git-bzr (http://github.com/kfish/git-bzr) HEAD, bazaar 2.1 and fastimport 0.9.0 dev

$ git bzr push $remote
11:04:16 Starting import ...
Could not import marks file .git/bzr-git/$remote-bzr-map - not importing marks
ABORT: exception occurred processing commit :3410
bzr: ERROR: exceptions.KeyError: ':3408'

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", line 853, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", line 1055, in run_bzr
    ret = run(*run_argv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", line 661, in run_argv_aliases
    return self.run_direct(**all_cmd_args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", line 665, in run_direct
    return self._operation.run_simple(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/cleanup.py", line 122, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/cleanup.py", line 156, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/plugins/fastimport/__init__.py", line 384, in run
    params, verbose, user_map=user_map)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/plugins/fastimport/__init__.py", line 111, in _run
    return proc.process(p.iter_commands)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/plugins/fastimport/processor.py", line 95, in process
    self._process(command_iter)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/plugins/fastimport/processors/generic_processor.py", line 283, in _process
    processor.ImportProcessor._process(self, command_iter)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/plugins/fastimport/processor.py", line 117, in _process
    handler(self, cmd)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/plugins/fastimport/processors/generic_processor.py", line 493, in commit_handler
    handler.process()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/plugins/fastimport/processor.py", line 200, in process
    self.pre_process_files()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/plugins/fastimport/bzr_commit_handler.py", line 582, in pre_process_files
    super(InventoryDeltaCommitHandler, self).pre_process_files()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/p...

[Ricky (rickyr)]

This thread seems to have gone quiet, but the attached push_back_once.patch seems to do the trick for me.

[Vasil Dimov (vasild)]

I am also experiencing the exact same problem.

bzr 2.1.1
fastimport 0.9.0
git 1.7.0.6

[Vasil Dimov (vasild)]

Very easy to repeat, please fix this:

hag:~$ bzr --version
Bazaar (bzr) 2.1.1
  Python interpreter: /usr/local/bin/python2.6 2.6.4
  Python standard library: /usr/local/lib/python2.6
  Platform: FreeBSD-8.0-STABLE-amd64-64bit-ELF
  bzrlib: /usr/local/lib/python2.6/site-packages/bzrlib
  Bazaar configuration: /usr/home/vd-dev/.bazaar
  Bazaar log file: /usr/home/vd-dev/.bzr.log

Copyright 2005-2010 Canonical Ltd.
http://bazaar-vcs.org/

bzr comes with ABSOLUTELY NO WARRANTY. bzr is free software, and
you may use, modify and redistribute it under the terms of the GNU
General Public License version 2 or later.

Bazaar is part of the GNU Project to produce a free operating system.

hag:~/.bazaar/plugins/fastimport$ bzr pull
Using saved parent location: bzr+ssh://bazaar.launchpad.net/~bzr/bzr-fastimport/trunk/
... No revisions to pull.
hag:~/.bazaar/plugins/fastimport$ date
Wed May 5 20:33:25 EEST 2010

hag:/tmp/t/g$ git --version
git version 1.7.0.6

hag:/tmp$ mkdir t
hag:/tmp$ cd t
hag:/tmp/t$ mkdir b
hag:/tmp/t$ cd b
hag:/tmp/t/b$ bzr init
... bzr output ...
hag:/tmp/t/b$ echo foo > f
hag:/tmp/t/b$ bzr add f
adding f
hag:/tmp/t/b$ bzr commit -m 'add f'
... bzr output ...
hag:/tmp/t/b$ echo bar >> f
hag:/tmp/t/b$ bzr commit -m 'add bar'
... bzr output ...
hag:/tmp/t/b$ cd ..
hag:/tmp/t$ mkdir g
hag:/tmp/t$ cd g
hag:/tmp/t/g$ git init
Initialized empty Git repository in /tmp/t/g/.git/
hag:/tmp/t/g$ git bzr add b_remote /tmp/t/b
+ gitbzr_version=1.2
+ return=
+ gitbzr_run add b_remote /tmp/t/b
+ '[' 3 -lt 2 ']'
+ cmd=add
+ shift
+ args='b_remote /tmp/t/b'
+ '[' xadd '!=' xhelp ']'
+ git rev-parse
+ '[' 0 -ne 0 ']'
++ git rev-parse --show-cdup
+ up=
+ '[' x == x ']'
+ up=.
+ cd .
+ case $cmd in
+ gitbzr_add b_remote /tmp/t/b
+ '[' 2 -lt 2 ']'
+ name=b_remote
+ location=/tmp/t/b
+ [[ -z b_remote /tmp/t/b ]]
++ git remote show
++ grep -q '"b_remote"'
+ '[' -n '' ']'
++ git config git-bzr.b_remote.url
+ '[' -n '' ']'
+ bzr info -q /tmp/t/b
+ git config git-bzr.b_remote.location /tmp/t/b
+ echo -e 'Bazaar branch b_remote added.\nYou can fetch it with '\''git bzr fetch b_remote'\''.'
Bazaar branch b_remote added.
You can fetch it with 'git bzr fetch b_remote'.
hag:/tmp/t/g$ git bzr fetch b_remote
+ gitbzr_version=1.2
+ return=
+ gitbzr_run fetch b_remote
+ '[' 2 -lt 2 ']'
+ cmd=fetch
+ shift
+ args=b_remote
+ '[' xfetch '!=' xhelp ']'
+ git rev-parse
+ '[' 0 -ne 0 ']'
++ git rev-parse --show-cdup
+ up=
+ '[' x == x ']'
+ up=.
+ cd .
+ case $cmd in
+ gitbzr_fetch b_remote
+ '[' 1 -lt 1 ']'
++ gitbzr_git_directory
+++ git rev-parse --git-dir
++ gitdir=.git
++ '[' -e .git ']'
+++ realpath .git
++ gitdir=/tmp/t/g/.git
++ echo /tmp/t/g/.git
+ gitdir=/tmp/t/g/.git
+ remote=b_remote
+ shift
+ args=
+ '[' -z b_remote ']'
+ get_location b_remote
+ '[' 1 -lt 1 ']'
+ remote=b_remote
+ shift
+ args=
++ git config git-bzr.b_remote.location
+ l=/tmp/t/b
+ '[' -z /tmp/t/b ']'
+ return=/tmp/t/b
+ location=/tmp/t/b
+ git_map=/tmp/t/g/.git/bzr-git/b_remote-git-map
+ bzr_map=/tmp/t/g/.git/bzr-git/b_remote-bzr-map
+ '[' '!' -f /tmp/t/g/.git/bzr-git/b_remote-git-map -a '!' -f /tmp/t/g/.git/bzr-git/b_remote-bzr-map ']'
+ echo 'There doesn'\''t seem to be an existing refma...

[Vasil Dimov (vasild)]

Just the commands to reproduce (without the output):

(assuming you do not have a directory called /tmp/t)

cd /tmp
mkdir t
cd t
mkdir b
cd b
bzr init
echo foo > f
bzr add f
bzr commit -m 'add f'
echo bar >> f
bzr commit -m 'add bar'
cd ..
mkdir g
cd g
git init
git config git-bzr.b_remote.location /tmp/t/b
mkdir -p /tmp/t/g/.git/bzr-git
bzr fast-export --export-marks=/tmp/t/g/.git/bzr-git/b_remote-bzr-map --git-branch=b_remote /tmp/t/b |git fast-import --export-marks=/tmp/t/g/.git/bzr-git/b_remote-git-map
git branch b_local b_remote
git checkout b_local
echo baz >> f
git commit -a -m 'add baz'
git fast-export --import-marks=/tmp/t/g/.git/bzr-git/b_remote-git-map --export-marks=/tmp/t/g/.git/bzr-git/b_remote-git-map HEAD |(cd /tmp/t/b && bzr fast-import --import-marks=/tmp/t/g/.git/bzr-git/b_remote-bzr-map --export-marks=/tmp/t/g/.git/bzr-git/b_remote-bzr-map -)

[Vasil Dimov (vasild)]

Hello? Anybody hear me?

[Silex (pvaucher)]

Yes, anyone knows what causes the bug?
Would it be as easy as fixing some internal format bzr/git uses?

[Vasil Dimov (vasild)]

Silex, yes - there is an inconsistency in the bzr export/import parsing of "marks".
If I do

sed 's/^([0-9]+) /:\1 /'

on /tmp/t/g/.git/bzr-git/b_remote-bzr-map before the last bzr command then it works.

The file is created by bzr without leading ":" while on import it expects the marks to have leading ":".

[Miklos Vajna (vmiklos)]

On Wed, May 19, 2010 at 02:34:58PM -0000, Vasil Dimov <email address hidden> wrote:
> Silex, yes - there is an inconsistency in the bzr export/import parsing of "marks".
> If I do
>
> sed 's/^([0-9]+) /:\1 /'
>
> on /tmp/t/g/.git/bzr-git/b_remote-bzr-map before the last bzr command
> then it works.
>
> The file is created by bzr without leading ":" while on import it
> expects the marks to have leading ":".

In fact it could be whatever, just be consistent. ;)

Though git-fast-import/export uses the ":" prefix, so keeping that in
bzr would be a good idea as well.

[Silex (pvaucher)]

So the fix is trivial and nobody fixed it in git-bzr repository? Shame on you! :)

That said, I'll propose a patch once I have a little bit of time on my hands, if the proposed substition above works.

[Matthew Brett (matthew-brett)]

It looks like the main problem then is the lack of colons ':' in the bzr mark file. I patched my python git-bzr script with a colon filter routine, and it seems to work for basic use (example script in http://github.com/matthew-brett/git-bzr README.

[Matthew Brett (matthew-brett)]

Just for information, here's an expanded version of Ryan's original script showing that, we need to add colons to the marks read by bzr fast-import, but the colons need to be absent for bzr fast-export to work:

mkdir bzr-temp
cd bzr-temp
bzr init
touch a
bzr add a
bzr commit -m "yep"
cd ..
mkdir git-temp
cd git-temp
git init
bzr fast-export --export-marks=.git/bzr-upstream --git-branch=bzr/upstream ../bzr-temp > .git/temp1
git fast-import --export-marks=.git/git-upstream < .git/temp1
git checkout -b local_branch bzr/upstream
touch b
git add b
git commit -m "yepppppppppp"
git fast-export --import-marks=.git/git-upstream --export-marks=.git/git-upstream HEAD > .git/temp2
sed 's/^\([0-9][0-9]* \)/:\1/' .git/bzr-upstream > .git/bzr-upstream-colons
cd ..
cd bzr-temp
bzr fast-import --import-marks=../git-temp/.git/bzr-upstream-colons --export-marks=../git-temp/.git/bzr-upstream-colons ../git-temp/.git/temp2
echo 'something' > c
bzr add c
bzr commit -m 'adding c'
cd ..
cd git-temp
bzr fast-export --import-marks=.git/bzr-upstream --export-marks=.git/bzr-upstream --git-branch=bzr/upstream ../bzr-temp > .git/temp3
git fast-import --export-marks=.git/git-upstream --import-marks=.git/git-upstream < .git/temp3

[Vasil Dimov (vasild)]

Hello?

[Jelmer Vernooij (jelmer)]

Has anybody tried Ian's new-marks-format branch ?

[Vasil Dimov (vasild)]

Hi Jelmer, where is that branch? I will try it.

This is not yet fixed in bzr+ssh://bazaar.launchpad.net/~bzr/bzr-fastimport/trunk/

Thanks!

[Jelmer Vernooij (jelmer)]

On Tue, 2010-07-27 at 19:42 +0000, Vasil Dimov wrote:
> Hi Jelmer, where is that branch? I will try it.
>
> This is not yet fixed in bzr+ssh://bazaar.launchpad.net/~bzr/bzr-
> fastimport/trunk/

It's linked on the bug page.

Cheers,

Jelmer

[Vasil Dimov (vasild)]

I can't find it from here: https://bugs.launchpad.net/bzr-fastimport

Please post the exact URL.

[Gabriel Filion (gabriel-filion)]

Jelmer:

I just tried out the branch lp:~ian-clatworthy/bzr-fastimport/new-marks-format

and I still get an error. but it's a different one this time:

bzr: ERROR: exceptions.KeyError: 0

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 853, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 1055, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 661, in run_argv_aliases
    return self.run_direct(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 665, in run_direct
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/cleanup.py", line 122, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/cleanup.py", line 156, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/home/gabster/.bazaar/plugins/fastimport/__init__.py", line 339, in run
    params, verbose)
  File "/home/gabster/.bazaar/plugins/fastimport/__init__.py", line 108, in _run
    return proc.process(p.iter_commands)
  File "/home/gabster/.bazaar/plugins/fastimport/processor.py", line 95, in process
    self._process(command_iter)
  File "/home/gabster/.bazaar/plugins/fastimport/processors/generic_processor.py", line 275, in _process
    processor.ImportProcessor._process(self, command_iter)
  File "/home/gabster/.bazaar/plugins/fastimport/processor.py", line 109, in _process
    self.pre_process()
  File "/home/gabster/.bazaar/plugins/fastimport/processors/generic_processor.py", line 134, in pre_process
    self.cache_mgr.revision_ids = mark_info[0]
KeyError: 0

bzr 2.1.2 on python 2.5.4 (Linux-2.6.32-5-openvz-686-i686-with-debian-squeeze-sid)
arguments: ['/usr/bin/bzr', 'fast-import', '--import-marks=/home/gabster/test/lolo/.git/bzr/map/master-bzr', '--export-marks=/home/gabster/test/lolo/.git/bzr/map/master-bzr', '-', '/home/gabster/test/lolo/.git/bzr/repo/master']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_CA.UTF-8'
plugins:
  bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [2.1.0]
  fastimport /home/gabster/.bazaar/plugins/fastimport [0.9.0dev]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [2.1.2]
  netrc_credential_store /usr/lib/python2.5/site-packages/bzrlib/plugins/netrc_credential_store [2.1.2]
  news_merge /usr/lib/python2.5/site-packages/bzrlib/plugins/news_merge [2.1.2]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.
No new revisions to push.

In the file processors/generic_processor.py, the marks_file.import_marks() function returns a dictionary of marks as {revno: rev-hash} affected to mark_info, so retrieving mark_info[0] is wrong.
also, cache_mgr is expecting a dictionary of revisions in revsion_ids. I tried changing line 134 to "self.cache_m...

[Jelmer Vernooij (jelmer)]

On Thu, 2010-07-29 at 09:33 +0000, Vasil Dimov wrote:
> I can't find it from here: https://bugs.launchpad.net/bzr-fastimport

> Please post the exact URL.

That's the overall bug page, it's linked on the top of the page for this
specific bug, https://bugs.launchpad.net/bugs/347729

[Gabriel Filion (gabriel-filion)]

OK, I was really missing something: I wasn't giving 'git fast-export' a ref to export..

So, I confirm that with the modification that I mentioned on Ian Clatworthy's "new-marks-format" branch, bzr fast-import is working correctly.

To make it easier to review, here's a diff of the modification on the branch:

=== modified file 'processors/generic_processor.py'
--- processors/generic_processor.py 2009-08-21 16:29:36 +0000
+++ processors/generic_processor.py 2010-08-08 00:32:44 +0000
@@ -131,7 +131,7 @@
         if self.params.get("import-marks") is not None:
             mark_info = marks_file.import_marks(self.params.get("import-marks"))
             if mark_info is not None:
- self.cache_mgr.revision_ids = mark_info[0]
+ self.cache_mgr.revision_ids = mark_info
             self.skip_total = False
             self.first_incremental_commit = True
         else:

[Gabriel Filion (gabriel-filion)]

I continued playing with Ian's branch + my patch and I encountered another very similar bug. I'll report it here since it's touching on the same subject (namely, changing the mark_file format to a consistant one between import and export). I'm attaching a crash report from bzr that shows what is happening.

I guess that not all cases were modified to honor the new format.

[Gabriel Filion (gabriel-filion)]

I performed some more testing and I was able to reproduce scenarios that are working and those that hit the bug that I mentioned in my last message.

What is working:
  git fast-export master | bzr fast-import - local/dst/branch # no marks file imported
  the same scenario but with --import-mark as a marks file that was previously created by bzr fast-import

What is not working:
  using a marks file with bzr fast-import that was created by bzr fast-export

I found the following differences in contents in the mark files between the scenarios:

Here's an extract of a marks file generated by the bzr fast-export command:

:1 guillaume@methradon-20090519143909-2hr1pvscdy9yr0mv
:2 <email address hidden>
:3 <email address hidden>
:4 <email address hidden>
:5 <email address hidden>
:6 <email address hidden>

Here's an extract of a marks file generated by the bzr fast-import command (sorted out, it comes out in another random order):

::1 guillaume@methradon-20090519143909-2hr1pvscdy9yr0mv
::2 <email address hidden>
::3 <email address hidden>
::4 <email address hidden>
::5 <email address hidden>
::6 <email address hidden>

So there's still (in Ian's branch) a difference between mark files generated (and expected) by both commands.

[Gabriel Filion (gabriel-filion)]

I've put the status back to Confirmed since trunk is still affected, the "push_back_once.patch" file does not really fix the issue and Ian's branch nearly does so but there's still something with it.

I also tried replaying Ian's modification about changing the marks file format on top of trunk and the situation is the same. I get the error in generic_processor.py. After applying my patch on top of trunk + Ian's patch, I now get an error of the kind (originating from bzrlib?):

bzr: ERROR: exceptions.AttributeError: 'InMemoryGraphIndex' object has no attribute '_find_ancestors'

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 853, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1055, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 661, in run_argv_aliases
    return self.run_direct(**all_cmd_args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 665, in run_direct
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/cleanup.py", line 122, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/cleanup.py", line 156, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/home/gabster/.bazaar/plugins/fastimport/__init__.py", line 384, in run
    params, verbose, user_map=user_map)
  File "/home/gabster/.bazaar/plugins/fastimport/__init__.py", line 111, in _run
    return proc.process(p.iter_commands)
  File "/home/gabster/.bazaar/plugins/fastimport/processor.py", line 95, in process
    self._process(command_iter)
  File "/home/gabster/.bazaar/plugins/fastimport/processors/generic_processor.py", line 283, in _process
    processor.ImportProcessor._process(self, command_iter)
  File "/home/gabster/.bazaar/plugins/fastimport/processor.py", line 117, in _process
    handler(self, cmd)
  File "/home/gabster/.bazaar/plugins/fastimport/processors/generic_processor.py", line 493, in commit_handler
    handler.process()
  File "/home/gabster/.bazaar/plugins/fastimport/processor.py", line 208, in process
    self.post_process_files()
  File "/home/gabster/.bazaar/plugins/fastimport/bzr_commit_handler.py", line 607, in post_process_files
    self._get_inventories)
  File "/home/gabster/.bazaar/plugins/fastimport/revision_store.py", line 374, in load_using_delta
    [(r,) for r in rev.parent_ids])
  File "/usr/lib/python2.6/dist-packages/bzrlib/knit.py", line 1196, in get_known_graph_ancestry
    parent_map, missing_keys = self._index.find_ancestry(keys)
  File "/usr/lib/python2.6/dist-packages/bzrlib/knit.py", line 3129, in find_ancestry
    return self._graph_index.find_ancestry(keys, 0)
  File "/usr/lib/python2.6/dist-packages/bzrlib/index.py", line 1405, in find_ancestry
    search_keys = index._find_ancestors(search_keys,
AttributeError: 'InMemoryGraphIndex' object has no attribute '_find_ancestors'

bzr 2.1.2 on python 2.6.6 (Linux-2.6.32-5-openvz-686-i686-with-debian-squeeze-sid)
arguments: ['/usr/bin/bzr', 'fast-import', '--e...

[Jelmer Vernooij (jelmer)]

Gabriel, that's bug 541626.

[Jelmer Vernooij (jelmer)]

I've merged Ian's branch and Gabriel's improvement to it. If you find any other bugs, please open separate bug reports.

[Jelmer Vernooij (jelmer)]

The _find_ancestors issue is bug 541626.