Comment 4 for bug 191731

Revision history for this message
Rexbard (john-klinger) wrote : Re: Memory problems prevent branch of large svn repositories

Tried a branch of all revisions, "bzr branch svn+http://...", using bzr 1.2 and it still ran out of memory after 3.5 hrs and close to 9200 revision_ids were added. However, as I mentioned earlier, there is a workaround, as I am able to create a branch by pulling about 1-2k revisions at a time, "bzr branch -r1000 svn+http://... project; cd project; bzr pull -r2000; ...".

I still can't tell if the problem is in bzr, bzr-svn, or the subversion bindings. Let me know if there is any diagnostics or tests you'd like me to execute.

The tracebacks when it dies are inconsistent, since they report when the process runs out of memory, not who's taking it all. Here's the tracebacks from the full branch attempt (a mix of console and .bzr.log output):

bzr: ERROR: bzrlib.errors.KnitCorrupt: Knit <bzrlib.knit._PackAccess object at 0x2aaab52e5190> corrupt: While reading {svn-v3-trunk0:3282bf91-d132-0410-b1ef-d2f8e70e1952:trunk:9168} got MemoryError()

13224.904 Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/bzrlib/commands.py", line 834, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib64/python2.4/site-packages/bzrlib/commands.py", line 790, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib64/python2.4/site-packages/bzrlib/commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib64/python2.4/site-packages/bzrlib/builtins.py", line 908, in run
    accelerator_tree=accelerator_tree)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/remote.py", line 75, in sprout
    result_repo.fetch(repo, revision_id=revision_id)
  File "/usr/lib64/python2.4/site-packages/bzrlib/repository.py", line 949, in fetch
    return inter.fetch(revision_id=revision_id, pb=pb, find_ghosts=find_ghosts)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/fetch.py", line 716, in fetch
    self._fetch_switch(needed, pb, lhs_parent)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/fetch.py", line 673, in _fetch_switch
    reporter.finish_report(pool)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/errors.py", line 110, in convert
    return unbound(*args, **kwargs)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/transport.py", line 271, in finish_report
    self._baton, pool)
  File "/usr//lib/svn-python/libsvn/ra.py", line 783, in svn_ra_reporter2_invoke_finish_report
    return apply(_ra.svn_ra_reporter2_invoke_finish_report, args)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/fetch.py", line 386, in close_file
    self._store_file(self.file_id, lines, self.file_parents)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/svn/fetch.py", line 467, in _store_file
    file_weave.add_lines(self.revid, parents, lines)
  File "/usr/lib64/python2.4/site-packages/bzrlib/versionedfile.py", line 122, in add_lines
    left_matching_blocks, nostore_sha, random_id, check_content)
  File "/usr/lib64/python2.4/site-packages/bzrlib/knit.py", line 925, in _add_lines
    parent_texts, left_matching_blocks, nostore_sha, random_id)
  File "/usr/lib64/python2.4/site-packages/bzrlib/knit.py", line 993, in _add
    left_matching_blocks)
  File "/usr/lib64/python2.4/site-packages/bzrlib/knit.py", line 856, in _merge_annotations
    reference_content = self._get_content(parents[0], parent_texts)
  File "/usr/lib64/python2.4/site-packages/bzrlib/knit.py", line 905, in _get_content
    text_map, contents_map = self._get_content_maps([version_id])
  File "/usr/lib64/python2.4/site-packages/bzrlib/knit.py", line 1085, in _get_content_maps
    record_map = self._get_record_map(version_ids)
  File "/usr/lib64/python2.4/site-packages/bzrlib/knit.py", line 1049, in _get_record_map
    for component_id, content, digest in \
  File "/usr/lib64/python2.4/site-packages/bzrlib/knit.py", line 2496, in read_records_iter
    content, digest = self._parse_record(version_id, data)
  File "/usr/lib64/python2.4/site-packages/bzrlib/knit.py", line 2401, in _parse_record
    "While reading {%s} got %s(%s)"
KnitCorrupt: Knit <bzrlib.knit._PackAccess object at 0x2aaab52e5190> corrupt: While reading {svn-v3-trunk0:3282bf91-d132-0410-b1ef-d2f8e70e1952:trunk:9168} got MemoryError()

bzr 1.2.0 on python 2.4.3.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'branch', '--verbose', 'svn+http://192.168.4.23/repos/rocc/branches/rdo_0_1', 'bzr.rdo_0_1']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bzrtools /usr/lib/python2.4/site-packages/bzrlib/plugins/bzrtools [1.0.0]
  gtk /usr/lib/python2.4/site-packages/bzrlib/plugins/gtk [0.93.0]
  launchpad /usr/lib64/python2.4/site-packages/bzrlib/plugins/launchpad [unknown]
  multiparent /usr/lib64/python2.4/site-packages/bzrlib/plugins/multiparent.py [unknown]
  svn /usr/lib/python2.4/site-packages/bzrlib/plugins/svn [0.4.7]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

13241.506 return code 4