fast-export using existing marks is broken

[Ben Hutchings]

Running 'brz fast-export' with the --export-marks (or --marks) option results in an internal error traceback:

$ mkdir brz-test
$ cd brz-test/
$ brz init
Created a standalone tree (format: 2a)
$ touch foo
$ brz add foo
adding foo
$ brz commit -m foo
bzr: warning: The commit message is a file name: "foo".
(use --file "foo" to take commit message from that file)
Committing to: /home/bwh/tmp/brz-test/
added foo
Committed revision 1.
$ brz fast-export --git-branch=trunk . ../brz-test-fastexport --export-marks=../brz-test-marks
16:47:58 Calculating the revisions to include ...
16:47:58 Starting export of 1 revisions ...
brz: ERROR: AttributeError: 'int' object has no attribute 'lstrip'

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/breezy/commands.py", line 1022, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/breezy/commands.py", line 1208, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python3/dist-packages/breezy/commands.py", line 762, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python3/dist-packages/breezy/commands.py", line 787, in run
    return class_run(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/breezy/plugins/fastimport/cmds.py", line 507, in run
    return exporter.run()
  File "/usr/lib/python3/dist-packages/breezy/plugins/fastimport/exporter.py", line 282, in run
    self._save_marks()
  File "/usr/lib/python3/dist-packages/breezy/plugins/fastimport/exporter.py", line 330, in _save_marks
    marks_file.export_marks(self.export_marks_file, revision_ids)
  File "/usr/lib/python3/dist-packages/breezy/plugins/fastimport/marks_file.py", line 81, in export_marks
    f.write(':%s %s\n' % (mark.lstrip(b':').decode('utf-8'),
AttributeError: 'int' object has no attribute 'lstrip'

brz 3.1.0 on python 3.8.5 (Linux-4.19.0-9-amd64-x86_64-with-glibc2.29)
arguments: ['/usr/bin/brz', 'fast-export', '--git-branch=trunk', '--export-
    marks=../brz-test-marks', '.', './brz-test-fastexport']
plugins: bash_completion[3.1.0], changelog_merge[3.1.0],
    commitfromnews[3.1.0], cvs[3.1.0], darcs[3.1.0], email[3.1.0],
    fastimport[3.1.0], flake8[unknown], fossil[3.1.0], github[3.1.0],
    gitlab[3.1.0], hg[3.1.0], launchpad[3.1.0], mtn[3.1.0],
    netrc_credential_store[3.1.0], news_merge[3.1.0], po_merge[3.1.0],
    propose[3.1.0], quilt[unknown], repodebug[3.1.0], rewrite[unknown],
    stats[3.1.0], svn[3.1.0], upload[3.1.0], weave_fmt[3.1.0],
    zsh_completion[3.1.0]
encoding: 'ascii', fsenc: 'utf-8', lang: 'C'

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/brz/+filebug
    including this traceback and a description of the problem.

A different error message is shown when using the --marks option and a previously generated marks file:

    13:47:46 Calculating the revisions to include ...
    13:47:46 Starting export of 446 revisions ...
    brz: ERROR: KeyError: b'null:'

    Traceback (most recent call last):
      File "/usr/lib/python3.8/site-packages/breezy/commands.py", line 1022, in exception_to_return_code
        return the_callable(*args, **kwargs)
      File "/usr/lib/python3.8/site-packages/breezy/commands.py", line 1208, in run_bzr
        ret = run(*run_argv)
      File "/usr/lib/python3.8/site-packages/breezy/commands.py", line 762, in run_argv_aliases
        return self.run(**all_cmd_args)
      File "/usr/lib/python3.8/site-packages/breezy/commands.py", line 787, in run
        return class_run(*args, **kwargs)
      File "/usr/lib/python3.8/site-packages/breezy/plugins/fastimport/cmds.py", line 507, in run
        return exporter.run()
      File "/usr/lib/python3.8/site-packages/breezy/plugins/fastimport/exporter.py", line 277, in run
        self.emit_commits(interesting)
      File "/usr/lib/python3.8/site-packages/breezy/plugins/fastimport/exporter.py", line 266, in emit_commits
        self.emit_commit(revobj, self.ref, trees[parent], trees[revid])
    KeyError: b'null:'

    brz 3.1.0 on python 3.8.3 (Linux-4.19.0-9-cloud-amd64-x86_64-with-glibc2.2.5)
    arguments: ['/usr/bin/brz', 'fast-export', '--git-branch=trunk', '/srv/lorry-
        working-area/delta_pytz/bzr/trunk', '/srv/lorry-working-
        area/delta_pytz/fast-exporttrunk', '--marks=/srv/lorry-working-
        area/delta_pytz/git/marks.bzr']
    plugins: bash_completion[3.1.0], changelog_merge[3.1.0],
        commitfromnews[3.1.0], cvs[3.1.0], darcs[3.1.0], email[3.1.0],
        fastimport[3.1.0], flake8[unknown], fossil[3.1.0], github[3.1.0],
        gitlab[3.1.0], hg[3.1.0], launchpad[3.1.0], mtn[3.1.0],
        netrc_credential_store[3.1.0], news_merge[3.1.0], po_merge[3.1.0],
        propose[3.1.0], quilt[unknown], repodebug[3.1.0], rewrite[unknown],
        stats[3.1.0], svn[3.1.0], upload[3.1.0], weave_fmt[3.1.0],
        zsh_completion[3.1.0]
    encoding: 'utf-8', fsenc: 'utf-8', lang: 'C.UTF-8'

    *** Bazaar has encountered an internal error. This probably indicates a
        bug in Bazaar. You can help us fix it by filing a bug report at
            https://bugs.launchpad.net/brz/+filebug
        including this traceback and a description of the problem.

[Ben Hutchings (ben-hutchings-y)]

I realise now that I've been testing two different versions.

The first failure is seen in 3.1.0. I believe it was fixed by https://code.launchpad.net/~jelmer/brz/marks-export/+merge/386706 but that fix is not present in any release yet.

The second failure is seen in r7580. I'll re-test with the top of the brz-3.1 branch.

I can close this and open a new report for the second failure if that's helpful.

[Ben Hutchings (ben-hutchings-y)]

The second failure is seen in r7616. I'll try to come up with a simple reproducer.

[Ben Hutchings (ben-hutchings-y)]

Reproducer for the second failure (KeyError: b'null:'):

# use Bazaar for the initial checkout, though it might work with Breezy too
bzr checkout lp:libmemcached
# use Breezy for the export, with attached marks file
brz fast-export --git-branch=trunk libmemcached - --marks libmemcached-marks >/dev/null

[Sebastian Geiger (lanoxx)]

I tried to migrate a bazaar repository to git using bzr-git-mass-convert (https://code.launchpad.net/~ubuntu-core-dev/+git/bzr-git-mass-convert/+ref/master) today and ran into this issue. I tried both on Ubuntu 20.04 and with a virtual machine using 20.10 (groovy beta). I am not sure how to continue now.

[Stdedos (stdedos)]

@Sebastian, are you sure the process did not end? :/

The output is indeed disturbing, but I did manage to extract a small repo to git alright (I only cared about the commits, so I got them)

With that said, fixing the bugs I think is always a good idea - since the software has to be used,

[void-spark (void-spark)]

Ran into this again today on debian bookworm. Seems it can't find a key which comes from a constant breezy.revision.NULL_REVISION. The actual trees map is empty. Bit of logging/debugging:

            for revid in chunk:
                revobj = history[revid]
                if len(revobj.parent_ids) == 0:
                    self.note("PARENT A")
                    parent = breezy.revision.NULL_REVISION
                else:
                    self.note("PARENT B")
                    parent = revobj.parent_ids[0]
                self.note("REVOBJ")
                self.note(revobj)
                self.note("SELF")
                self.note(self.ref)
                self.note("parent")
                self.note(parent)
                self.note("trees")
                self.note(trees)
                self.emit_commit(revobj, self.ref, trees[parent], trees[revid])

16:34:35 Calculating the revisions to include ...
16:34:35 Starting export of 1355 revisions ...
16:34:35 PARENT A
16:34:35 REVOBJ
16:34:35 <Revision id <email address hidden>'>
16:34:35 SELF
16:34:35 b'refs/heads/toykeeper'
16:34:35 parent
16:34:35 b'null:'
16:34:35 trees
16:34:35 {}
brz: ERROR: KeyError: b'null:'

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/breezy/commands.py", line 1036, in exception_to_return_code
    return the_callable(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/breezy/commands.py", line 1228, in run_bzr
    ret = run(*run_argv)
          ^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/breezy/commands.py", line 779, in run_argv_aliases
    return self.run(**all_cmd_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/breezy/commands.py", line 804, in run
    return class_run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/breezy/plugins/fastimport/cmds.py", line 505, in run
    return exporter.run()
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/breezy/plugins/fastimport/exporter.py", line 277, in run
    self.emit_commits(interesting)
  File "/usr/lib/python3/dist-packages/breezy/plugins/fastimport/exporter.py", line 266, in emit_commits
    self.emit_commit(revobj, self.ref, trees[parent], trees[revid])
                                       ~~~~~^^^^^^^^
KeyError: b'null:'

I used to get around it by using a very old version on debian buster, but for some reason buster takes forever to build it's python packages or something now..