Error: Newly created pack file has delta references to items not in its repository

[Peter Bienstman]

I get the following error when checking out an svn repository and then subsequently pushing it to a fresh location in launchpad.

Also, I was wondering if the time required for the checkout (more than 20 minutes) is normal

> time bzr co https://mnemosyne-proj.svn.sourceforge.net/svnroot/mnemosyne-proj/trunk trunk
Initialising Subversion metadata cache in /home/pbienst/.bazaar/svn-cache/e5e6b78b-db40-0410-9517-b98c64f8d2c1
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 1432.07 s

> cd trunk
> bzr push bzr+ssh://<email address hidden>/~peter-bienstman/mnemosyne-proj/trunk
bzr: ERROR: bzrlib.errors.BzrCheckError: Internal check failed: Newly created pack file <bzrlib.repofmt.pack_repo.NewPack object at 0x7f1b56892350> has delta references to items not in its repository:
{'texts': [('2@e5e6b78b-db40-0410-9517-b98c64f8d2c1:trunk:mnemosyne%2FTODO',
            'pbienst@amar-sin-20081118192140-egevc8uy9ep9q87m')]}

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 896, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 842, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 542, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 981, in run
    use_existing_dir=use_existing_dir)
  File "/usr/lib/python2.5/site-packages/bzrlib/push.py", line 105, in _show_push_branch
    revision_id=revision_id, stacked_on=stacked_on)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 224, in clone_on_transport
    result_repo.fetch(local_repo, revision_id=revision_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 1118, in fetch
    find_ghosts=find_ghosts)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2985, in fetch
    return self._pack(self.source, self.target, revision_ids)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2992, in _pack
    revision_ids).pack()
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 655, in pack
    return self._create_pack_from_packs()
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 825, in _create_pack_from_packs
    new_pack._check_references()
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 348, in _check_references
    % (self, pformat(missing_items)))
BzrCheckError: Internal check failed: Newly created pack file <bzrlib.repofmt.pack_repo.NewPack object at 0x7f1b56892350> has delta references to items not in its repository:
{'texts': [('2@e5e6b78b-db40-0410-9517-b98c64f8d2c1:trunk:mnemosyne%2FTODO',
            'pbienst@amar-sin-20081118192140-egevc8uy9ep9q87m')]}

bzr 1.12 on python 2.5.2 (linux2)
arguments: ['/usr/bin/bzr', 'push', 'bzr+ssh://<email address hidden>/~peter-bienstman/mnemosyne-proj/trunk']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
  netrc_credential_store /usr/lib/python2.5/site-packages/bzrlib/plugins/netrc_credential_store [unknown]
  svn /usr/lib/python2.5/site-packages/bzrlib/plugins/svn [0.5.2]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

[Jelmer Vernooij (jelmer)]

On Thu, 2009-02-26 at 15:43 +0000, Peter Bienstman wrote:
> Public bug reported:
>
> I get the following error when checking out an svn repository and then
> subsequently pushing it to a fresh location in launchpad.
>
> Also, I was wondering if the time required for the checkout (more than
> 20 minutes) is normal
This seems mainly caused by the server side, the sf.net repository isn't
exactly the fastest in the world.

> > time bzr co https://mnemosyne-proj.svn.sourceforge.net/svnroot/mnemosyne-proj/trunk trunk
> Initialising Subversion metadata cache in /home/pbienst/.bazaar/svn-cache/e5e6b78b-db40-0410-9517-b98c64f8d2c1
> CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
> Wall time: 1432.07 s
>
> > cd trunk
> > bzr push bzr+ssh://<email address hidden>/~peter-bienstman/mnemosyne-proj/trunk
> bzr: ERROR: bzrlib.errors.BzrCheckError: Internal check failed: Newly created pack file <bzrlib.repofmt.pack_repo.NewPack object at 0x7f1b56892350> has delta references to items not in its repository:
> {'texts': [('2@e5e6b78b-db40-0410-9517-b98c64f8d2c1:trunk:mnemosyne%2FTODO',
> 'pbienst@amar-sin-20081118192140-egevc8uy9ep9q87m')]}
>
> Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 896, in run_bzr_catch_errors
> return run_bzr(argv)
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 842, in run_bzr
> ret = run(*run_argv)
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 542, in run_argv_aliases
> return self.run(**all_cmd_args)
> File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 981, in run
> use_existing_dir=use_existing_dir)
> File "/usr/lib/python2.5/site-packages/bzrlib/push.py", line 105, in _show_push_branch
> revision_id=revision_id, stacked_on=stacked_on)
> File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 224, in clone_on_transport
> result_repo.fetch(local_repo, revision_id=revision_id)
> File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 1118, in fetch
> find_ghosts=find_ghosts)
> File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
> result = unbound(self, *args, **kwargs)
> File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2985, in fetch
> return self._pack(self.source, self.target, revision_ids)
> File "/usr/lib/python2.5/s...

[Peter Bienstman (peter-bienstman)]

>> Also, I was wondering if the time required for the checkout (more than
>> 20 minutes) is normal
>This seems mainly caused by the server side, the sf.net repository isn't
>exactly the fastest in the world.

Still,
time svn co https://mnemosyne-proj.svn.sourceforge.net/svnroot/mnemosyne-proj/trunk trunk
Checked out revision 715.
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 13.54 s

Wouldn't it be possible to first get the svn data in 1 batch like this, and then do all the required processing locally?

[Jelmer Vernooij (jelmer)]

On Thu, 2009-02-26 at 18:10 +0000, Peter Bienstman wrote:
> >> Also, I was wondering if the time required for the checkout (more than
> >> 20 minutes) is normal
> >This seems mainly caused by the server side, the sf.net repository isn't
> >exactly the fastest in the world.
>
> Still,
> time svn co https://mnemosyne-proj.svn.sourceforge.net/svnroot/mnemosyne-proj/trunk trunk
> Checked out revision 715.
> CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
> Wall time: 13.54 s
>
> Wouldn't it be possible to first get the svn data in 1 batch like this,
> and then do all the required processing locally?
these two operations do different things.

svn co only retrieves the last revision, bzr co retrieves the full
 history (all revisions). there is no way in svn to retrieve all history at once.

You can retrieve just the last revision with bzr as well, by running
"bzr co --lightweight <url>". Of course this has other drawbacks, since
more operations will then have to contact the server.

--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

[Peter Bienstman (peter-bienstman)]

That seems to take just as long:

 time bzr co --lightweight https://mnemosyne-proj.svn.sourceforge.net/svnroot/mnemosyne-proj/trunk trunk
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 1142.20 s

Shall I open a separate bug report on this?

[Jelmer Vernooij (jelmer)]

On Thu, 2009-02-26 at 19:10 +0000, Peter Bienstman wrote:
> time bzr co --lightweight
> https://mnemosyne-proj.svn.sourceforge.net/svnroot/mnemosyne-proj/trunk trunk
Ah, this is a corner case. It has to retrieve the file properties set by
bzr-svn 0.4.x in order to determine the file ids.

Generally this will be a lot faster, but in this particular case it's
not because bzr-svn 0.4.x was used heavily on the repository.

If you didn't have those bzr-svn 0.4.x file properties set in the
repository, it wouldn't need to contact the repository for the file
properties of most of the revisions in the history. The revision
properties that bzr-svn 0.5.x sets don't have this problem.

If you run this command again it should be faster though, since bzr-svn
should be caching the file id map.

I agree it's a bit unfortunate this is so slow, but this should not be
an issue anymore in the future, when bzr-svn 0.4.x is no longer used.

Cheers,

Jelmer

--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

[Jelmer Vernooij (jelmer)]

  affects bzr
  status confirmed

  affects bzr-svn
  status invalid

  duplicate 328904
--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>