The bzr server repository got corrupted and nobody can pull or push changes to it. Running "bzr check" and "bzr reconcile" shows the following messages:
[11:51:11][~/trunk-r16]$ bzr merge
Merging from remembered submit location bzr+ssh://<email address hidden>/bzr/client-facebook/trunk/
bzr: ERROR: bzrlib.errors.ErrorFromSmartServer: Error received from smart server: ('ShortReadvError', 'b22a569ce92096b22330c933808dc3b1.tix', '0', '772', '60')
Traceback (most recent call last):
File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 853, in exception_to_return_code
return the_callable(*args, **kwargs)
File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 1055, in run_bzr
ret = run(*run_argv)
File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 661, in run_argv_aliases
return self.run_direct(**all_cmd_args)
File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 665, in run_direct
return self._operation.run_simple(*args, **kwargs)
File "/Library/Python/2.6/site-packages/bzrlib/cleanup.py", line 122, in run_simple
self.cleanups, self.func, *args, **kwargs)
File "/Library/Python/2.6/site-packages/bzrlib/cleanup.py", line 156, in _do_with_cleanups
result = func(*args, **kwargs)
File "/Library/Python/2.6/site-packages/bzrlib/builtins.py", line 3794, in run
location, revision, remember, possible_transports, pb)
File "/Library/Python/2.6/site-packages/bzrlib/builtins.py", line 3923, in _get_merger_from_branch
other_revision_id, base_revision_id, other_branch, base_branch)
File "/Library/Python/2.6/site-packages/bzrlib/merge.py", line 363, in from_revision_ids
merger.set_other_revision(other, other_branch)
File "/Library/Python/2.6/site-packages/bzrlib/merge.py", line 505, in set_other_revision
self._maybe_fetch(other_branch, self.this_branch, self.other_rev_id)
File "/Library/Python/2.6/site-packages/bzrlib/merge.py", line 522, in _maybe_fetch
target.fetch(source, revision_id)
File "/Library/Python/2.6/site-packages/bzrlib/decorators.py", line 194, in write_locked
result = unbound(self, *args, **kwargs)
File "/Library/Python/2.6/site-packages/bzrlib/branch.py", line 593, in fetch
pb=pb)
File "/Library/Python/2.6/site-packages/bzrlib/repository.py", line 1704, in fetch
find_ghosts=find_ghosts, fetch_spec=fetch_spec)
File "/Library/Python/2.6/site-packages/bzrlib/decorators.py", line 194, in write_locked
result = unbound(self, *args, **kwargs)
File "/Library/Python/2.6/site-packages/bzrlib/repository.py", line 3438, in fetch
pb=pb, find_ghosts=find_ghosts)
File "/Library/Python/2.6/site-packages/bzrlib/fetch.py", line 83, in __init__
self.__fetch()
File "/Library/Python/2.6/site-packages/bzrlib/fetch.py", line 109, in __fetch
self._fetch_everything_for_search(search)
File "/Library/Python/2.6/site-packages/bzrlib/fetch.py", line 137, in _fetch_everything_for_search
stream, from_format, [])
File "/Library/Python/2.6/site-packages/bzrlib/repository.py", line 4279, in insert_stream
return self._locked_insert_stream(stream, src_format, is_resume)
File "/Library/Python/2.6/site-packages/bzrlib/repository.py", line 4316, in _locked_insert_stream
substream)
File "/Library/Python/2.6/site-packages/bzrlib/knit.py", line 1599, in insert_record_stream
for record in stream:
File "/Library/Python/2.6/site-packages/bzrlib/versionedfile.py", line 1665, in read
for bytes in self._bytes_iterator:
File "/Library/Python/2.6/site-packages/bzrlib/smart/repository.py", line 571, in iter_substream_bytes
for record in self.iter_pack_records:
File "/Library/Python/2.6/site-packages/bzrlib/smart/repository.py", line 561, in iter_stream_decoder
for bytes in self.byte_stream:
File "/Library/Python/2.6/site-packages/bzrlib/smart/message.py", line 338, in read_streamed_body
_translate_error(self._body_error_args)
File "/Library/Python/2.6/site-packages/bzrlib/smart/message.py", line 361, in _translate_error
raise errors.ErrorFromSmartServer(error_tuple)
ErrorFromSmartServer: Error received from smart server: ('ShortReadvError', 'b22a569ce92096b22330c933808dc3b1.tix', '0', '772', '60')
bzr 2.1.1 on python 2.6.1 (Darwin-10.4.0-i386-64bit)
arguments: ['/usr/local/bin/bzr', 'merge']
encoding: 'UTF-8', fsenc: 'utf-8', lang: 'en_US.UTF-8'
plugins:
bzrtools /Library/Python/2.6/site-packages/bzrlib/plugins/bzrtools [2.1.0]
email /Library/Python/2.6/site-packages/bzrlib/plugins/email [unknown]
explorer /Library/Python/2.6/site-packages/bzrlib/plugins/explorer [1.0.1]
extmerge /Users/dbanks/.bazaar/plugins/extmerge [unknown]
keychain /Library/Python/2.6/site-packages/bzrlib/plugins/keychain [0.1.0]
launchpad /Library/Python/2.6/site-packages/bzrlib/plugins/launchpad [2.1.1]
netrc_credential_store /Library/Python/2.6/site-packages/bzrlib/plugins/netrc_credential_store [2.1.1]
news_merge /Library/Python/2.6/site-packages/bzrlib/plugins/news_merge [2.1.1]
qbzr /Library/Python/2.6/site-packages/bzrlib/plugins/qbzr [0.18.4]
rebase /Library/Python/2.6/site-packages/bzrlib/plugins/rebase [0.5.5]
svn /Library/Python/2.6/site-packages/bzrlib/plugins/svn [1.0.2]
upload /Library/Python/2.6/site-packages/bzrlib/plugins/upload [1.0.0dev]
xmloutput /Users/dbanks/.bazaar/plugins/xmloutput [0.8.4]
*** Bazaar has encountered an internal error. This probably indicates a
bug in Bazaar. You can help us fix it by filing a bug report at
https://bugs.launchpad.net/bzr/+filebug
including this traceback and a description of the problem.
[11:51:19][~/trunk-r16]$
build@dev-001:/bzr/client-facebook/trunk$ bzr check
Checking branch at 'file:///glass/sfw/bzr/repo/client-facebook/release/'.
Checking branch at 'file:///glass/sfw/bzr/repo/client-facebook/trunk/'.
Checking branch at 'file:///glass/sfw/bzr/repo/client-facebook/work/'.
No working tree found at specified location.
Checking repository at 'file:///glass/sfw/bzr/repo/client-facebook/'.
bzr: ERROR: bzrlib.errors.ShortReadvError: readv() read 60 bytes rather than 772 bytes at 0 for "b22a569ce92096b22330c933808dc3b1.tix"
Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 842, in exception_to_return_code
return the_callable(*args, **kwargs)
File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 1037, in run_bzr
ret = run(*run_argv)
File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 654, in run_argv_aliases
return self.run(**all_cmd_args)
File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 3145, in run
check_dwim(path, verbose, do_branch=branch, do_repo=repo, do_tree=tree)
File "/usr/lib/python2.5/site-packages/bzrlib/check.py", line 456, in check_dwim
check_repo=do_repo)
File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 138, in read_locked
result = unbound(self, *args, **kwargs)
File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2767, in check
check_repo=check_repo)
File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2771, in _check
result.check(callback_refs)
File "/usr/lib/python2.5/site-packages/bzrlib/check.py", line 102, in check
self.repository._check_inventories(self)
File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 1172, in _check_inventories
self._do_check_inventories(checker, bar)
File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 1223, in _do_check_inventories
for record in getattr(self, kind).check(keys=keys[kind]):
File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1405, in get_record_stream
ordering, include_delta_closure):
File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1416, in _get_remaining_record_stream
positions = self._get_components_positions(keys, allow_missing=True)
File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1166, in _get_components_positions
build_details = self._index.get_build_details(pending_components)
File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 3051, in get_build_details
for entry in entries:
File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 3079, in _get_entries
for node in self._graph_index.iter_entries(keys):
File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 1290, in iter_entries
for node in index.iter_entries(keys):
File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 635, in iter_entries
if self._nodes is None and len(keys) * 20 > self.key_count():
File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 744, in key_count
self._read_and_parse([_HEADER_READV])
File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 1170, in _read_and_parse
for offset, data in readv_data:
File "/usr/lib/python2.5/site-packages/bzrlib/transport/__init__.py", line 701, in _seek_and_read
c_offset.length, actual=len(data))
ShortReadvError: readv() read 60 bytes rather than 772 bytes at 0 for "b22a569ce92096b22330c933808dc3b1.tix"
bzr 2.0.4 on python 2.5.2 (Linux-2.6.24-26-xen-x86_64-with-debian-lenny-sid)
arguments: ['/usr/bin/bzr', 'check']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [2.0.0]
launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [2.0.4]
netrc_credential_store /usr/lib/python2.5/site-packages/bzrlib/plugins/netrc_credential_store [2.0.4]
xmloutput /usr/lib/python2.5/site-packages/bzrlib/plugins/xmloutput [0.8.3]
*** Bazaar has encountered an internal error. This probably indicates a
bug in Bazaar. You can help us fix it by filing a bug report at
https://bugs.launchpad.net/bzr/+filebug
including this traceback and a description of the problem.
build@dev-001:/bzr/client-facebook/trunk$ bzr reconcile
Reconciling branch file:///glass/sfw/bzr/repo/client-facebook/trunk/
revision_history ok.
Reconciling repository file:///glass/sfw/bzr/repo/client-facebook/
bzr: ERROR: exceptions.AssertionError: 772 59
Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 842, in exception_to_return_code
return the_callable(*args, **kwargs)
File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 1037, in run_bzr
ret = run(*run_argv)
File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 654, in run_argv_aliases
return self.run(**all_cmd_args)
File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 1597, in run
reconcile(dir)
File "/usr/lib/python2.5/site-packages/bzrlib/reconcile.py", line 53, in reconcile
reconciler.reconcile()
File "/usr/lib/python2.5/site-packages/bzrlib/reconcile.py", line 77, in reconcile
self._reconcile()
File "/usr/lib/python2.5/site-packages/bzrlib/reconcile.py", line 84, in _reconcile
self._reconcile_repository()
File "/usr/lib/python2.5/site-packages/bzrlib/reconcile.py", line 103, in _reconcile_repository
repo_reconciler = self.repo.reconcile(thorough=True)
File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
result = unbound(self, *args, **kwargs)
File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 2352, in reconcile
reconciler.reconcile()
File "/usr/lib/python2.5/site-packages/bzrlib/reconcile.py", line 200, in reconcile
self._reconcile_steps()
File "/usr/lib/python2.5/site-packages/bzrlib/reconcile.py", line 513, in _reconcile_steps
".reconcile", all_revisions, self.pb)
File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 2357, in _reconcile_pack
return packer.pack(pb)
File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 760, in pack
return self._create_pack_from_packs()
File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 923, in _create_pack_from_packs
self._copy_text_texts()
File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 1268, in _copy_text_texts
for node in text_nodes:
File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 1265, in iter_all_entries
for node in index.iter_all_entries():
File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 505, in iter_all_entries
self._buffer_all()
File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 445, in _buffer_all
_, _, _, trailers = self._parse_lines(lines, pos)
File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 1074, in _parse_lines
raise AssertionError("%s %s" % (self._size, pos))
AssertionError: 772 59
bzr 2.0.4 on python 2.5.2 (Linux-2.6.24-26-xen-x86_64-with-debian-lenny-sid)
arguments: ['/usr/bin/bzr', 'reconcile']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [2.0.0]
launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [2.0.4]
netrc_credential_store /usr/lib/python2.5/site-packages/bzrlib/plugins/netrc_credential_store [2.0.4]
xmloutput /usr/lib/python2.5/site-packages/bzrlib/plugins/xmloutput [0.8.3]
*** Bazaar has encountered an internal error. This probably indicates a
bug in Bazaar. You can help us fix it by filing a bug report at
https://bugs.launchpad.net/bzr/+filebug
including this traceback and a description of the problem.
ali@dev-001:/bzr/server/trunk$ bzr info -v
Repository branch (format: unnamed)
Location:
shared repository: /glass/sfw/bzr/repo/server
repository branch: .
Related branches:
parent branch: /glass/sfw/bzr/repo/server/release
Format:
control: Meta directory format 1
branch: Branch format 7
repository: Packs containing knits without subtree support
Branch history:
155 revisions
59 days old
first revision: Fri 2010-07-16 14:19:02 -0700
latest revision: Mon 2010-09-13 14:57:40 -0700
Repository:
7522 revisions
Through IRC channel, people think it might be the OS file system truncating the file: expected 772 bytes, but only 59 bytes found. The OS is Ubuntu 64-bit, and has been running fine for 4 months. No system level errors or logs can be found.
Our current work around is to rollback to daily backup.
Thanks
Thanks for attaching the index files.
This is a bit different from most other truncation situations because the files are clearly incomplete but non-zero.
Marking incomplete/low because it's possible this is a bzr bug but without dupes or more data it's going to be hard to localize it.
One interesting thing is that all of them have a header and then a blank line at the end. It's possible that means bzr was line buffering the output and it was interrupted after writing a few lines but not all of them. (But I wouldn't normally think that we would be line buffering files written over a Transport...)
I see you're accessing the branch over ssh. Do you know what OS and filesystem is on the server?