[fastimport 3.0.2] TypeError: startswith first arg must be bytes or a tuple of bytes, not str

[Yorigami Shion]

Tried to fast-import a git repository to brz but failed:(

```
root@first:/mnt/Git/McCLIM/McCLIM# git --version
git version 2.17.1
root@first:/mnt/Git/McCLIM/McCLIM# git fast-export --all --signed-tags=verbatim > project.fi
root@first:/mnt/Git/McCLIM/McCLIM# brz fast-import project.fi McCLIM.brz
brz: ERROR: AttributeError: 'NoneType' object has no attribute '_transport'

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/breezy-3.0.2-py2.7-linux-x86_64.egg/breezy/commands.py", line 1016, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/breezy-3.0.2-py2.7-linux-x86_64.egg/breezy/commands.py", line 1202, in run_bzr
    ret = run(*run_argv)
  File "/usr/local/lib/python2.7/dist-packages/breezy-3.0.2-py2.7-linux-x86_64.egg/breezy/commands.py", line 759, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/local/lib/python2.7/dist-packages/breezy-3.0.2-py2.7-linux-x86_64.egg/breezy/commands.py", line 784, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/breezy-3.0.2-py2.7-linux-x86_64.egg/breezy/cleanup.py", line 137, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/breezy-3.0.2-py2.7-linux-x86_64.egg/breezy/cleanup.py", line 166, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/breezy-3.0.2-py2.7-linux-x86_64.egg/breezy/plugins/fastimport/cmds.py", line 322, in run
    user_map=user_map)
  File "/usr/local/lib/python2.7/dist-packages/breezy-3.0.2-py2.7-linux-x86_64.egg/breezy/plugins/fastimport/cmds.py", line 51, in _run
    return proc.process(p.iter_commands)
  File "/usr/local/lib/python2.7/dist-packages/breezy-3.0.2-py2.7-linux-x86_64.egg/breezy/plugins/fastimport/processors/generic_processor.py", line 282, in process
    super(GenericProcessor, self)._process(command_iter)
  File "build/bdist.linux-x86_64/egg/fastimport/processor.py", line 78, in _process
    self.pre_process()
  File "/usr/local/lib/python2.7/dist-packages/breezy-3.0.2-py2.7-linux-x86_64.egg/breezy/plugins/fastimport/processors/generic_processor.py", line 147, in pre_process
    self._load_info_and_params()
  File "/usr/local/lib/python2.7/dist-packages/breezy-3.0.2-py2.7-linux-x86_64.egg/breezy/plugins/fastimport/processors/generic_processor.py", line 210, in _load_info_and_params
    repo_transport = self.repo.control_files._transport
AttributeError: 'NoneType' object has no attribute '_transport'

brz 3.0.2 on python 2.7.15 (Linux-5.3.0-23-generic-x86_64-with-
    Ubuntu-18.04-bionic)
arguments: ['/usr/local/bin/brz', 'fast-import', 'project.fi', 'McCLIM.brz']
plugins: bash_completion[3.0.2], changelog_merge[3.0.2],
    commitfromnews[3.0.2], cvs[3.0.2], darcs[3.0.2], email[3.0.2],
    fastimport[3.0.2], launchpad[3.0.2], mtn[3.0.2],
    netrc_credential_store[3.0.2], news_merge[3.0.2], po_merge[3.0.2],
    propose[3.0.2], repodebug[3.0.2], stats[3.0.2], upload[3.0.2],
    weave_fmt[3.0.2]
encoding: 'utf-8', fsenc: 'UTF-8', lang: 'C.UTF-8'

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/brz/+filebug
    including this traceback and a description of the problem.

```

[Peter J. Mello (roguescholar)]

Heh, turns out it doesn't work under Python3 either, though naturally the specific error is different. This is what I'm encountering

...

$ git fast-export --tag-of-filtered-object=rewrite --signed-tags=strip --all > debian.fi

$ brz fast-import debian.fi ../xtables-addons/debian
brz: ERROR: TypeError: startswith first arg must be bytes or a tuple of bytes, not str

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/breezy/commands.py", line 1016, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/breezy/commands.py", line 1202, in run_bzr
    ret = run(*run_argv)
  File "/usr/local/lib/python3.7/dist-packages/breezy/commands.py", line 759, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/local/lib/python3.7/dist-packages/breezy/commands.py", line 784, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/breezy/cleanup.py", line 137, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/breezy/cleanup.py", line 166, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/breezy/plugins/fastimport/cmds.py", line 304, in run
    info = self._generate_info(source)
  File "/usr/local/lib/python3.7/dist-packages/breezy/plugins/fastimport/cmds.py", line 336, in _generate_info
    return_code = proc.process(p.iter_commands)
  File "/usr/lib/python3/dist-packages/fastimport/processor.py", line 75, in process
    self._process(command_iter)
  File "/usr/lib/python3/dist-packages/fastimport/processor.py", line 87, in _process
    handler(self, cmd)
  File "/usr/lib/python3/dist-packages/fastimport/processors/info_processor.py", line 241, in reset_handler
    if cmd.ref.startswith('refs/tags/'):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

...

Breezy (brz) 3.0.2
Python interpreter: /usr/bin/python3.7 3.7.5
Python standard library: /usr/lib/python3.7
Platform: Linux-5.3.0-23-generic-x86_64-with-Ubuntu-19.10-eoan

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jelmer Vernooij (jelmer)]

On Sun, Dec 01, 2019 at 05:16:24AM -0000, Yorigami Shion wrote:
> Public bug reported:
>
> Tried to fast-import a git repository to brz but failed:(
>
> ```
> root@first:/mnt/Git/McCLIM/McCLIM# git --version
> git version 2.17.1
> root@first:/mnt/Git/McCLIM/McCLIM# git fast-export --all --signed-tags=verbatim > project.fi
> root@first:/mnt/Git/McCLIM/McCLIM# brz fast-import project.fi McCLIM.brz
> brz: ERROR: AttributeError: 'NoneType' object has no attribute '_transport'
There seem to be two separate issues here:

* lp:~jelmer/brz/fastimport-fix-2 should address the transport
  attribute

* python-fastimport assumes plain strings rather than bytestrings;
  this is fixed in python-fastimport master.

Jelmer

[Peter J. Mello (roguescholar)]

@jelmer,

I just compiled python-fastimport from git master (your GitHub repo) and the fastimport-fix-2 branch you linked in your comment. Prior to that I uninstalled Cython and reinstalled to try and eliminate any other variables. Sadly, while the error message has changed slightly, it hasn't abated. This is what it says now

...

brz: ERROR: TypeError: sequence item 0: expected str instance, bytes found

Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/commands.py", line 1016, in exception_to_return_code
  return the_callable(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/commands.py", line 1202, in run_bzr
  ret = run(*run_argv)
File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/commands.py", line 759, in run_argv_aliases
  return self.run(**all_cmd_args)
File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/commands.py", line 784, in run
  return self._operation.run_simple(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/cleanup.py", line 137, in run_simple
  self.cleanups, self.func, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/cleanup.py", line 166, in _do_with_cleanups
  result = func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/plugins/fastimport/cmds.py", line 304, in run
  info = self._generate_info(source)
File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/plugins/fastimport/cmds.py", line 336, in _generate_info
  return_code = proc.process(p.iter_commands)
File "/usr/local/lib/python3.7/dist-packages/fastimport/processor.py", line 75, in process
  self._process(command_iter)
File "/usr/local/lib/python3.7/dist-packages/fastimport/processor.py", line 91, in _process
  self.post_process()
File "/usr/local/lib/python3.7/dist-packages/fastimport/processors/info_processor.py", line 137, in
  post_process self.blobs.items(), len, _iterable_as_config_list)
File "/usr/local/lib/python3.7/dist-packages/fastimport/processors/info_processor.py", line 167, in
  _dump_stats_group value = verbose_formatter(value)
File "/usr/local/lib/python3.7/dist-packages/fastimport/processors/info_processor.py", line 298, in
_iterable_as_config_list
  return ", ".join(items)
TypeError: sequence item 0: expected str instance, bytes found

[Jelmer Vernooij (jelmer)]

Hi Peter,

What command are you running exactly? This is still a bug, but may not
happen if you disable verbose mode.

Jelmer

On Mon, Dec 02, 2019 at 06:33:39AM -0000, Peter J. Mello wrote:
> @jelmer,
>
> I just compiled python-fastimport from git master (your GitHub repo) and
> the fastimport-fix-2 branch you linked in your comment. Prior to that I
> uninstalled Cython and reinstalled to try and eliminate any other
> variables. Sadly, while the error message has changed slightly, it
> hasn't abated. This is what it says now
>
> ...
>
> brz: ERROR: TypeError: sequence item 0: expected str instance, bytes
> found
>
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/commands.py", line 1016, in exception_to_return_code
> return the_callable(*args, **kwargs)
> File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/commands.py", line 1202, in run_bzr
> ret = run(*run_argv)
> File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/commands.py", line 759, in run_argv_aliases
> return self.run(**all_cmd_args)
> File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/commands.py", line 784, in run
> return self._operation.run_simple(*args, **kwargs)
> File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/cleanup.py", line 137, in run_simple
> self.cleanups, self.func, *args, **kwargs)
> File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/cleanup.py", line 166, in _do_with_cleanups
> result = func(*args, **kwargs)
> File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/plugins/fastimport/cmds.py", line 304, in run
> info = self._generate_info(source)
> File "/usr/local/lib/python3.7/dist-packages/breezy-3.0.3.dev0-py3.7-linux-x86_64.egg/breezy/plugins/fastimport/cmds.py", line 336, in _generate_info
> return_code = proc.process(p.iter_commands)
> File "/usr/local/lib/python3.7/dist-packages/fastimport/processor.py", line 75, in process
> self._process(command_iter)
> File "/usr/local/lib/python3.7/dist-packages/fastimport/processor.py", line 91, in _process
> self.post_process()
> File "/usr/local/lib/python3.7/dist-packages/fastimport/processors/info_processor.py", line 137, in
> post_process self.blobs.items(), len, _iterable_as_config_list)
> File "/usr/local/lib/python3.7/dist-packages/fastimport/processors/info_processor.py", line 167, in
> _dump_stats_group value = verbose_formatter(value)
> File "/usr/local/lib/python3.7/dist-packages/fastimport/processors/info_processor.py", line 298, in
> _iterable_as_config_list
> return ", ".join(items)
> TypeError: sequence item 0: expected str instance, bytes found
>
> --
> You received this bug notification because you are subscribed to Breezy.
> https://bugs.launchpad.net/bugs/1854607
>
> Title:
> [fastimport 3.0.2] AttributeError: 'NoneType' object has no attribute
> '_transport'
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.ne...

[Peter J. Mello (roguescholar)]

Hi Jelmer,

First, thanks for helping me straighten out this issue. I'm working from a local clone of the xtables-addons git repo (git://git.code.sf.net/p/xtables-addons/xtables-addons). Originally I was trying to pipe the output of git fast-export [flags] to brz fast-import -, but I had absolutely no success with any of a multitude of permutations for the flags on both sides. Everything I've reported in this thread has been from attempts at the method tacitly suggested in the brz documentation, that of working from the git output in the form of a file. I created said file with the following command:

git fast-export --tag-of-filtered-object=rewrite --signed-tags=strip --all >debian.fi

I'm not all that familiar with the format specification for fast-export, but inspecting the contents of the file created by the above command shows an orderly arrangement of information that I have no reason to be suspicious of. I then attempted to process the debian.fi file with brz after moving into the folder I wanted the Bazaar repository to reside in with this command:

brz fast-import /path/to/debian.fi

I'm unaware of how one would go about "disabling verbose mode" on either the git or brz invocations; I was under the impression that verbose mode for both was an opt-in condition, not opt-out. I'll gladly test anything you might suggest, though.

Regards,
Peter

[Jelmer Vernooij (jelmer)]

Hi Peter,

It looks like this is another Python3 compatibility issue in
python-fastimport. I'll see if I can push a fix for it this evening.

Jelmer

On Tue, Dec 03, 2019 at 02:47:46PM -0000, Peter J. Mello wrote:
> First, thanks for helping me straighten out this issue. I'm working from
> a local clone of the xtables-addons git repo (git://git.code.sf.net/p
> /xtables-addons/xtables-addons). Originally I was trying to pipe the
> output of git fast-export [flags] to brz fast-import -, but I had
> absolutely no success with any of a multitude of permutations for the
> flags on both sides. Everything I've reported in this thread has been
> from attempts at the method tacitly suggested in the brz documentation,
> that of working from the git output in the form of a file. I created
> said file with the following command:
>
> git fast-export --tag-of-filtered-object=rewrite --signed-tags=strip
> --all >debian.fi
>
> I'm not all that familiar with the format specification for fast-export,
> but inspecting the contents of the file created by the above command
> shows an orderly arrangement of information that I have no reason to be
> suspicious of. I then attempted to process the debian.fi file with brz
> after moving into the folder I wanted the Bazaar repository to reside in
> with this command:
>
> brz fast-import /path/to/debian.fi
>
> I'm unaware of how one would go about "disabling verbose mode" on either
> the git or brz invocations; I was under the impression that verbose mode
> for both was an opt-in condition, not opt-out. I'll gladly test anything
> you might suggest, though.
>
> Regards,
> Peter
>
> ** Attachment added: "git fast-export output"
> https://bugs.launchpad.net/brz/+bug/1854607/+attachment/5309541/+files/debian.fi
>
> ** Summary changed:
>
> - [fastimport 3.0.2] AttributeError: 'NoneType' object has no attribute '_transport'
> + [fastimport 3.0.2] TypeError: startswith first arg must be bytes or a tuple of bytes, not str
>
> --
> You received this bug notification because you are subscribed to Breezy.
> https://bugs.launchpad.net/bugs/1854607
>
> Title:
> [fastimport 3.0.2] TypeError: startswith first arg must be bytes or a
> tuple of bytes, not str
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/brz/+bug/1854607/+subscriptions

[Jelmer Vernooij (jelmer)]

Hi Peter,

I've pushed a fix to the python-fastimport repository that allows me
to process the fastexport file you attached.

Jelmer

On Tue, Dec 03, 2019 at 02:47:46PM -0000, Peter J. Mello wrote:
> Hi Jelmer,
>
> First, thanks for helping me straighten out this issue. I'm working from
> a local clone of the xtables-addons git repo (git://git.code.sf.net/p
> /xtables-addons/xtables-addons). Originally I was trying to pipe the
> output of git fast-export [flags] to brz fast-import -, but I had
> absolutely no success with any of a multitude of permutations for the
> flags on both sides. Everything I've reported in this thread has been
> from attempts at the method tacitly suggested in the brz documentation,
> that of working from the git output in the form of a file. I created
> said file with the following command:
>
> git fast-export --tag-of-filtered-object=rewrite --signed-tags=strip
> --all >debian.fi
>
> I'm not all that familiar with the format specification for fast-export,
> but inspecting the contents of the file created by the above command
> shows an orderly arrangement of information that I have no reason to be
> suspicious of. I then attempted to process the debian.fi file with brz
> after moving into the folder I wanted the Bazaar repository to reside in
> with this command:
>
> brz fast-import /path/to/debian.fi
>
> I'm unaware of how one would go about "disabling verbose mode" on either
> the git or brz invocations; I was under the impression that verbose mode
> for both was an opt-in condition, not opt-out. I'll gladly test anything
> you might suggest, though.
>
> Regards,
> Peter
>
> ** Attachment added: "git fast-export output"
> https://bugs.launchpad.net/brz/+bug/1854607/+attachment/5309541/+files/debian.fi
>
> ** Summary changed:
>
> - [fastimport 3.0.2] AttributeError: 'NoneType' object has no attribute '_transport'
> + [fastimport 3.0.2] TypeError: startswith first arg must be bytes or a tuple of bytes, not str
>
> --
> You received this bug notification because you are subscribed to Breezy.
> https://bugs.launchpad.net/bugs/1854607
>
> Title:
> [fastimport 3.0.2] TypeError: startswith first arg must be bytes or a
> tuple of bytes, not str
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/brz/+bug/1854607/+subscriptions