IndexError pushing to knit repository over sftp

[Daniel Silverstone]

Pushing to a knit repository over sftp fails with the below message. Pushing the same branch to a knit repository on the filesystem does not. Ditto pushing to pack-0.92, 1.6 or 1.9 repositories over filesystem or sftp are all fine.

petitemort(bzr)% bzr push sftp://localhost/tmp/fooble/barble ~/dev-bzr/aranha[126]
Format <RepositoryFormatKnit1> for sftp://localhost/tmp/fooble/.bzr/ is deprecated - please use 'bzr upgrade' to get better performance
bzr: ERROR: exceptions.IndexError: list index out of range

Traceback (most recent call last):
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/commands.py", line 893, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/commands.py", line 839, in run_bzr
    ret = run(*run_argv)
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/commands.py", line 539, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/builtins.py", line 913, in run
    use_existing_dir=use_existing_dir)
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/push.py", line 105, in _show_push_branch
    revision_id=revision_id, stacked_on=stacked_on)
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/bzrdir.py", line 221, in clone_on_transport
    result_repo.fetch(local_repo, revision_id=revision_id)
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/repository.py", line 989, in fetch
    find_ghosts=find_ghosts)
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/repository.py", line 2578, in fetch
    pb=pb, find_ghosts=find_ghosts)
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/fetch.py", line 110, in __init__
    self.__fetch()
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/fetch.py", line 140, in __fetch
    self._fetch_everything_for_search(search, pp)
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/fetch.py", line 183, in _fetch_everything_for_search
    not self.to_repository._fetch_uses_deltas))
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/knit.py", line 1381, in insert_record_stream
    record, record.get_bytes_as(record.storage_kind)))
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/knit.py", line 228, in get_bytes
    [compression_parent], 'unordered', True).next()
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/knit.py", line 1248, in get_record_stream
    text_map, _ = self._get_content_maps(keys, non_local)
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/knit.py", line 999, in _get_content_maps
    record_map = self._get_record_map(keys, allow_missing=True)
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/knit.py", line 1118, in _get_record_map
    self._read_records_iter(records):
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/knit.py", line 1594, in _read_records_iter
    izip(iter(needed_records), raw_data):
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/knit.py", line 2369, in get_raw_records
    for pos, data in self._transport.readv(path, read_vector):
  File "/home/dsilvers/dev-bzr/bzr.dev/bzrlib/transport/sftp.py", line 298, in request_and_yield_offsets
    if data_chunks[idx][0] == cur_offset: # The data starts here
IndexError: list index out of range

bzr 1.10dev on python 2.5.2 (linux2)
arguments: ['/home/dsilvers/bin/bzr', 'push', 'sftp://localhost/tmp/fooble/barble']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_GB.UTF-8'
plugins:
  avahi /usr/lib/python2.5/site-packages/bzrlib/plugins/avahi [0.3dev]
  bookmarks /home/dsilvers/.bazaar/plugins/bookmarks [unknown]
  bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.6]
  dbus /usr/lib/python2.5/site-packages/bzrlib/plugins/dbus [unknown]
  email /home/dsilvers/.bazaar/plugins/email [unknown]
  gtk /usr/lib/python2.5/site-packages/bzrlib/plugins/gtk [0.95.0.final.1]
  launchpad /home/dsilvers/dev-bzr/bzr.dev/bzrlib/plugins/launchpad [unknown]
  loom /home/dsilvers/.bazaar/plugins/loom [1.4dev]
  pqm /home/dsilvers/.bazaar/plugins/pqm [1.4dev]
  search /home/dsilvers/.bazaar/plugins/search [1.7dev]
  svn /home/dsilvers/.bazaar/plugins/svn [0.4.14dev]
  transportstats /home/dsilvers/.bazaar/plugins/transportstats [unknown]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.
zsh: exit 4 bzr push sftp://localhost/tmp/fooble/barble

[John A Meinel (jameinel)]

I believe this is a regression in the sftp handling (caused be me), so I'll be looking into it.

[John A Meinel (jameinel)]

To start with, it would be nice to have the .bzr.log, as it includes a couple mutter() statements I would be interested.

Second, it probably primarily effects Knit repositories. That code path is only run when requests are made out-of-order, and pack repositories are better about doing things in whatever order is best for the repository (on-disk order), rather than requiring a specific order.

I'm still planning to fix it, I'm just trying to clarify the error and what can trigger it.

[John A Meinel (jameinel)]

The fix is pretty trivial, I'm attaching it (along with tests, etc)