wish format conversion was done separately, after fetching

[DimitryPolivaev]

The problem arises if I try to checkout a branch from remote source forge repository.
The branch currently has about 820 revisions, about 100 MB is transferred.
The remote repository has format rich-root-pack, the local repository has format 2a.

Version of my bzr client:
Bazaar (bzr) 2.0.1
  Python interpreter: /usr/bin/python 2.6.2
  Python standard library: /usr/lib/python2.6
  Platform: Linux-2.6.28-15-generic-i686-with-Ubuntu-9.04-jaunty
  bzrlib: /usr/lib/python2.6/dist-packages/bzrlib
  Bazaar configuration: /home/dimitry/.bazaar
  Bazaar log file: /home/dimitry/.bzr.log

I perform the following actions:

dimitry@dimitry-laptop:~$ mkdir bad-bazaar
dimitry@dimitry-laptop:~$ cd bad-bazaar/
dimitry@dimitry-laptop:~/bad-bazaar$ bzr init-repo --default-rich-root .
Shared repository with trees (format: 2a)
Location:
  shared repository: .
dimitry@dimitry-laptop:~/bad-bazaar$ bzr checkout bzr+ssh://<email address hidden>/bzrroot/freeplane/freeplane_program/release_branches/1_0_x
<email address hidden>'s password:
[#########/ ] 114914KB 322KB/s | Fetching revisions:Inserting stream

After 114914KB is read (it is almost the whole content to be transferred),
CPU load becomes high. After waiting a few minutes I get a message

Read from remote host freeplane.bzr.sourceforge.net: Connection reset by peeram

and later

bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.

The following messages are added to the .bzr.log file:

============== log fragment begin ===========
Tue 2009-10-20 19:17:46 +0200
0.078 bzr arguments: [u'init-repo', u'--default-rich-root', u'.']
0.085 looking for plugins in /home/dimitry/.bazaar/plugins
0.085 looking for plugins in /usr/lib/python2.6/dist-packages/bzrlib/plugins
0.230 encoding stdout as sys.stdout encoding 'UTF-8'
0.237 creating repository in file:///home/dimitry/bad-bazaar/.bzr/.
0.253 return code 0

Tue 2009-10-20 19:18:02 +0200
0.057 bzr arguments: [u'checkout', u'bzr+ssh://<email address hidden>/bzrroot/freeplane/freeplane_program/release_branches/1_0_x']
0.064 looking for plugins in /home/dimitry/.bazaar/plugins
0.064 looking for plugins in /usr/lib/python2.6/dist-packages/bzrlib/plugins
0.175 encoding stdout as sys.stdout encoding 'UTF-8'
0.230 ssh implementation is OpenSSH
11.123 creating branch <bzrlib.branch.BzrBranchFormat7 object at 0x997cb0c> in file:///home/dimitry/bad-bazaar/1_0_x/.bzr/
11.435 Using fetch logic to copy between RemoteRepository(bzr+ssh://<email address hidden>/bzrroot/freeplane/.bzr/)(<RemoteRepositoryFormat>) and CHKInventoryRepository('file:///home/dimitry/bad-bazaar/.bzr/repository/')(<RepositoryFormat2a>)
11.435 fetch up to rev {dimitry@dimitry-laptop-20091020055928-tkt6uauq29x4o3aa}
485.064 Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1037, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 654, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 1336, in run
    accelerator_tree, hardlink)
  File "/usr/lib/python2.6/dist-packages/bzrlib/branch.py", line 1330, in create_checkout
    checkout_branch.pull(self, stop_revision=revision_id)
  File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/branch.py", line 948, in pull
    possible_transports=possible_transports, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/branch.py", line 3194, in pull
    _override_hook_target=_override_hook_target)
  File "/usr/lib/python2.6/dist-packages/bzrlib/branch.py", line 3071, in pull
    overwrite=overwrite, graph=graph)
  File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/branch.py", line 896, in update_revisions
    overwrite, graph)
  File "/usr/lib/python2.6/dist-packages/bzrlib/branch.py", line 3014, in update_revisions
    self.target.fetch(self.source, stop_revision)
  File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/branch.py", line 579, in fetch
    pb=pb)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 1695, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 3413, in fetch
    pb=pb, find_ghosts=find_ghosts)
  File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 81, in __init__
    self.__fetch()
  File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 107, in __fetch
    self._fetch_everything_for_search(search)
  File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 135, in _fetch_everything_for_search
    stream, from_format, [])
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 4238, in insert_stream
    return self._locked_insert_stream(stream, src_format, is_resume)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 4278, in _locked_insert_stream
    substream, src_serializer)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 4366, in _extract_and_insert_inventories
    for record in substream:
  File "/usr/lib/python2.6/dist-packages/bzrlib/versionedfile.py", line 1604, in read
    for bytes in self._bytes_iterator:
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/repository.py", line 570, in iter_substream_bytes
    for record in self.iter_pack_records:
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/repository.py", line 560, in iter_stream_decoder
    for bytes in self.byte_stream:
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/message.py", line 336, in read_streamed_body
    self._read_more()
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/message.py", line 286, in _read_more
    "Unexpected end of message. "
ConnectionReset: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.

485.065 return code 3

============== log fragment end ===========

There is a workaround:
if I use the same repository format as on the source repository

bzr init-repo --rich-root-pack .
there is no time the CPU load is high and the same checkout works fine

============== log fragment begin ===========

Tue 2009-10-20 19:52:55 +0200
0.087 bzr arguments: [u'init-repo', u'--rich-root-pack', u'.']
0.103 looking for plugins in /home/dimitry/.bazaar/plugins
0.103 looking for plugins in /usr/lib/python2.6/dist-packages/bzrlib/plugins
0.242 encoding stdout as sys.stdout encoding 'UTF-8'
0.249 creating repository in file:///home/dimitry/bad-bazaar/.bzr/.
0.285 return code 0

Tue 2009-10-20 19:53:11 +0200
0.041 bzr arguments: [u'checkout', u'bzr+ssh://<email address hidden>/bzrroot/freeplane/freeplane_program/release_branches/1_0_x']
0.048 looking for plugins in /home/dimitry/.bazaar/plugins
0.048 looking for plugins in /usr/lib/python2.6/dist-packages/bzrlib/plugins
0.159 encoding stdout as sys.stdout encoding 'UTF-8'
0.217 ssh implementation is OpenSSH
10.313 creating branch <bzrlib.branch.BzrBranchFormat7 object at 0x8c5cb0c> in file:///home/dimitry/bad-bazaar/1_0_x/.bzr/
10.618 Using fetch logic to copy between RemoteRepository(bzr+ssh://<email address hidden>/bzrroot/freeplane/.bzr/)(<RemoteRepositoryFormat>) and KnitPackRepository('file:///home/dimitry/bad-bazaar/.bzr/repository/')(<RepositoryFormatKnitPack4>)
10.619 fetch up to rev {dimitry@dimitry-laptop-20091020055928-tkt6uauq29x4o3aa}
277.519 trying to create missing lock '/home/dimitry/bad-bazaar/1_0_x/.bzr/checkout/dirstate'
277.519 opening working tree '/home/dimitry/bad-bazaar/1_0_x'
280.244 return code 0
============== log fragment end ===========

[Matt Nordhoff (mnordhoff)]

Converting from rich-root-pack to 2a on the fly takes some effort. If
SourceForge's bzr server is anything like their svn server, it's
probably just taking up enough time or resources that SourceForge kills
the process or times out or something.

Of course, it's possible bzr is doing something dumb that exacerbates
this... Maybe someone more knowledgeable than me should weigh in.

[DimitryPolivaev (dpolivaev)]

You are right, we switched to bazaar because of poor performance of SF SVN. Yes, it looks like SourceForge times out durung the converting is running.

On the other side I suppose that this time it is not a SourceForge problem because the converting is running locally.
SourceForge still has an older version of bazaar, it known nothing about format 2a.
The message "Read from remote host freeplane.bzr.sourceforge.net: Connection reset by peeram" is displayed during the convertion is running, CPU load is high, and the following error message comes with significant delay. Therefore I still hope you investigate the reasons of the time out. I think If bazaar started the converting after everything was read and saved in a temporarily file and the connection was closed, it could solve the problem.

Dimitry

[Martin Pool (mbp)]

> If bazaar started the converting after everything was read and saved in a temporarily file and the connection was closed, it could solve the problem.

That could be interesting.

As a workaround you can do this yourself by having a mirror in the same format as the remote, pulling in to that, then into your real repository.