memory error on trying to checkout large subversion repository

[Martin Steigerwald]

I am trying to check out a subversion repository with about 870 revisions. One of them sports a large 400MB ISO file which has been deleted afterwards (cause it does not really belong into subversion anyway). The error may or may not be related to that large file.

I get this MemoryError:

ms@mango> bzr checkout svn+https://some.url
bzr: ERROR: exceptions.MemoryError:

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 834, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 790, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.4/site-packages/bzrlib/builtins.py", line 1012, in run
    accelerator_tree, hardlink)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/branch.py", line 193, in create_checkout
    return self._create_heavyweight_checkout(to_location, revision_id, hardlink=hardlink)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/branch.py", line 146, in _create_heavyweight_checkout
    checkout_branch.pull(self, stop_revision=revision_id)
  File "/usr/lib/python2.4/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.4/site-packages/bzrlib/branch.py", line 1714, in pull
    run_hooks=run_hooks)
  File "/usr/lib/python2.4/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.4/site-packages/bzrlib/branch.py", line 1513, in pull
    self.update_revisions(source, stop_revision, overwrite=overwrite)
  File "/usr/lib/python2.4/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.4/site-packages/bzrlib/branch.py", line 1465, in update_revisions
    self.fetch(other, stop_revision)
  File "/usr/lib/python2.4/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.4/site-packages/bzrlib/branch.py", line 266, in fetch
    pb=nested_pb)
  File "/usr/lib/python2.4/site-packages/bzrlib/repository.py", line 949, in fetch
    return inter.fetch(revision_id=revision_id, pb=pb, find_ghosts=find_ghosts)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/fetch.py", line 704, in fetch
    self._fetch_switch(needed, pb, lhs_parent)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/fetch.py", line 661, in _fetch_switch
    reporter.finish_report(pool)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/errors.py", line 117, in convert
    return unbound(*args, **kwargs)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/transport.py", line 280, in finish_report
    self._baton, pool)
  File "/var/lib/python-support/python2.4/libsvn/ra.py", line 738, in svn_ra_reporter2_invoke_finish_report
    return apply(_ra.svn_ra_reporter2_invoke_finish_report, args)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/fetch.py", line 365, in close_file
    lines = osutils.split_lines(self.file_stream.read())
MemoryError

bzr 1.3 on python 2.4.4.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'checkout', 'svn+https://some.url']
encoding: 'ISO-8859-15', fsenc: 'ISO-8859-15', lang: 'de_DE@euro'
plugins:
  bzrtools /usr/lib/python2.4/site-packages/bzrlib/plugins/bzrtools [1.3.0]
  launchpad /usr/lib/python2.4/site-packages/bzrlib/plugins/launchpad [unknown]
  rebase /usr/lib/python2.4/site-packages/bzrlib/plugins/rebase [0.3.0]
  svn /usr/lib/python2.4/site-packages/bzrlib/plugins/svn [0.4.9]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

This is the memory situation after the error has happened:

ms@mango> free -m
             total used free shared buffers cached
Mem: 1009 431 578 0 0 142
-/+ buffers/cache: 288 720
Swap: 1906 449 1456

mango:~# apt-show-versions | egrep "(bzr|subversion)"
bzr-rebase/unknown uptodate 0.3-1
bzr/etch uptodate 1.3-1~bpo40+1
bzr-svn/unknown uptodate 0.4.9-1
subversion/etch uptodate 1.4.2dfsg1-2
subversion-tools/etch uptodate 1.4.2dfsg1-2
python-subversion/etch uptodate 1.4.2dfsg1-2
bzrtools/unknown uptodate 1.3.0-2

Version "unknown" are packages from lenny, bzr itself is the latest backport from backports.org.

I can try this on my notebook with bzr 1.4, python 2.5 as soon as I have a compatible bzr-svn in sid or experimental.

[Jelmer Vernooij (jelmer)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

  status triaged
  importance low

This is a known issue - there is not really an easy way to fix this
because the delta received from Subversion is not translatable to the
delta Bazaar uses internally. This requires us to create a completely
snapshot and feed that to Bazaar. So a file may be held in memory two
or three times at the same moment.

Cheers,

Jelmer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSCtCNgy0JeEGD2blAQJsYAP/TK/yxtr1DGRad2Spd4hsfhMEzNo4FK4H
Fj6awvaR7fEU+oAilnHgntsD1tW9GNfVjSH6oz+8SUeBDUWy8Bz02jSp08J9AQOv
8zP9CMBKhCQRyFfZNFEYV8BX2ihr+Ax10cUzYubbYSAyPOroaMML1CpVCGvzFSwf
8JnPiVS3bH4=
=Mo27
-----END PGP SIGNATURE-----