ShortReadvError on index file

Bug #413430 reported by dbmcclain on 2009-08-14
136
This bug affects 18 people
Affects Status Importance Assigned to Milestone
Bazaar
High
Unassigned
Breezy
High
Unassigned

Bug Description

Summary:

You may encounter a message like this:

bzr: ERROR: bzrlib.errors.ShortReadvError: readv() read 268 bytes rather than 4096 bytes at 171764 for "d0580144782295bf6f36ded8f1415669.tix"

(This is distinct from the similar bug 413430 where the filename is ends in 'pack').

As far as we know, this is always caused by the file being truncated on disk outside of bzr's control. ext4 seems very prone to do this when a machine crashes or is abruptly halted.

However, it is a bug in bzr that the message is opaque and that there's no easy way out.

The shortest workaround if you hit this case is:

1- make a new repository directory
2- branch your main branches (trunk etc) into it, either from a backup or another copy of those branches; this will repopulate most of history
3- you should then be able to branch particular newly-changed branches into the new repository too

As a fix for this we could do any of:

1- try to force files to disk after writing
2- teach check to detect this situation and rebuild index files
...

bash-3.2$ bzr commit -m "Mach compiler improvements, Bfly recv takes an optional argument for the received message"
Committing to: /Volumes/Repository/srv/bzr/Lispworks/
modified Butterfly/Source/Keep-Alive.lisp
modified Butterfly/Source/Standard-Server.lisp
modified Butterfly/Source/bfly-mailbox.lisp
modified Butterfly/Source/bfly-message.lisp
modified Butterfly/Source/bfly-socket2.lisp
modified Butterfly/Source/bfly-spawn.lisp
modified Butterfly/Source/fru-replace-code.lisp
modified Okeanos/lock-daemon.lisp
modified Okeanos/remote-access.lisp
modified tools/useful-macros/match-macro-ex-opt.lisp

Traceback (most recent call last):
  File "/Library/Python/2.5/site-packages/bzrlib/commands.py", line 729, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/Library/Python/2.5/site-packages/bzrlib/commands.py", line 924, in run_bzr
    ret = run(*run_argv)
  File "/Library/Python/2.5/site-packages/bzrlib/commands.py", line 560, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/Library/Python/2.5/site-packages/bzrlib/builtins.py", line 2937, in run
    exclude=safe_relpath_files(tree, exclude))
  File "/Library/Python/2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/Library/Python/2.5/site-packages/bzrlib/workingtree_4.py", line 226, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/Library/Python/2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/Library/Python/2.5/site-packages/bzrlib/mutabletree.py", line 228, in commit
    *args, **kwargs)
  File "/Library/Python/2.5/site-packages/bzrlib/commit.py", line 397, in commit
    self.branch.repository, new_revno, self.rev_id)
  File "/Library/Python/2.5/site-packages/bzrlib/branch.py", line 821, in import_last_revision_info
    self.repository.fetch(source_repo, revision_id=revid)
  File "/Library/Python/2.5/site-packages/bzrlib/repository.py", line 1553, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/Library/Python/2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/Library/Python/2.5/site-packages/bzrlib/repository.py", line 3514, in fetch
    return self._pack(self.source, self.target, revision_ids)
  File "/Library/Python/2.5/site-packages/bzrlib/repository.py", line 3520, in _pack
    revision_ids).pack()
  File "/Library/Python/2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 753, in pack
    return self._create_pack_from_packs()
  File "/Library/Python/2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 943, in _create_pack_from_packs
    new_pack._check_references()
  File "/Library/Python/2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 189, in _check_references
    index.iter_entries(external_refs))
  File "/Library/Python/2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 188, in <genexpr>
    k for (idx, k, v, r) in
  File "/Library/Python/2.5/site-packages/bzrlib/index.py", line 1258, in iter_entries
    for node in index.iter_entries(keys):
  File "/Library/Python/2.5/site-packages/bzrlib/index.py", line 626, in iter_entries
    self._lookup_keys_via_location, self._size, keys))
  File "/Library/Python/2.5/site-packages/bzrlib/bisect_multi.py", line 50, in bisect_multi_bytes
    search_results = content_lookup(search_keys)
  File "/Library/Python/2.5/site-packages/bzrlib/index.py", line 769, in _lookup_keys_via_location
    self._read_and_parse(readv_ranges)
  File "/Library/Python/2.5/site-packages/bzrlib/index.py", line 1138, in _read_and_parse
    for offset, data in readv_data:
  File "/Library/Python/2.5/site-packages/bzrlib/transport/__init__.py", line 701, in _seek_and_read
    c_offset.length, actual=len(data))
ShortReadvError: readv() read 268 bytes rather than 4096 bytes at 171764 for "d0580144782295bf6f36ded8f1415669.tix"

bzr 1.16.1 on python 2.5.1 (darwin)
arguments: ['/usr/local/bin/bzr', 'commit', '-m', 'Mach compiler improvements, Bfly recv takes an optional argument for the received message']
encoding: 'UTF-8', fsenc: 'utf-8', lang: 'en_US.UTF-8'
plugins:
  bzrtools /Library/Python/2.5/site-packages/bzrlib/plugins/bzrtools [1.16]
  email /Library/Python/2.5/site-packages/bzrlib/plugins/email [unknown]
  extmerge /Library/Python/2.5/site-packages/bzrlib/plugins/extmerge [unknown]
  launchpad /Library/Python/2.5/site-packages/bzrlib/plugins/launchpad [1.16.1]
  loom /Library/Python/2.5/site-packages/bzrlib/plugins/loom [1.4dev]
  netrc_credential_store /Library/Python/2.5/site-packages/bzrlib/plugins/netrc_credential_store [1.16.1]
  qbzr /Library/Python/2.5/site-packages/bzrlib/plugins/qbzr [0.11]
  rebase /Library/Python/2.5/site-packages/bzrlib/plugins/rebase [0.5.1]
  search /Library/Python/2.5/site-packages/bzrlib/plugins/search [1.7dev]
  svn /Library/Python/2.5/site-packages/bzrlib/plugins/svn [0.6.2]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

The on-disk index file is apparently truncated. Could you please attach it to this bug and we can see if that is true?

summary: - Bug during commit to lab repository
+ ShortReadvError on tix file
Changed in bzr:
status: New → Incomplete
importance: Undecided → Medium
Download full text (6.5 KiB)

Hello Martin,

I would attach it for you, if I knew where it was and what it is
named. I'm not a BZR expert, just a devoted developer, and I use BZR
to keep safe copies of all my work and to distribute it to my other
workstations.

Cheers,

Dr. David McClain
<email address hidden>

On Aug 14, 2009, at 00:29 AM, Martin Pool wrote:

> The on-disk index file is apparently truncated. Could you please
> attach
> it to this bug and we can see if that is true?
>
> ** Summary changed:
>
> - Bug during commit to lab repository
> + ShortReadvError on tix file
>
> ** Changed in: bzr
> Status: New => Incomplete
>
> ** Changed in: bzr
> Importance: Undecided => Medium
>
> --
> ShortReadvError on tix file
> https://bugs.launchpad.net/bugs/413430
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Bazaar Version Control System: Incomplete
>
> Bug description:
> bash-3.2$ bzr commit -m "Mach compiler improvements, Bfly recv takes
> an optional argument for the received message"
> Committing to: /Volumes/Repository/srv/bzr/Lispworks/
> modified Butterfly/Source/Keep-Alive.lisp
> modified Butterfly/Source/Standard-Server.lisp
> modified Butterfly/Source/bfly-mailbox.lisp
> modified Butterfly/Source/bfly-message.lisp
> modified Butterfly/Source/bfly-socket2.lisp
> modified Butterfly/Source/bfly-spawn.lisp
> modified Butterfly/Source/fru-replace-code.lisp
> modified Okeanos/lock-daemon.lisp
> modified Okeanos/remote-access.lisp
> modified tools/useful-macros/match-macro-ex-opt.lisp
> bzr: ERROR: bzrlib.errors.ShortReadvError: readv() read 268 bytes
> rather than 4096 bytes at 171764 for
> "d0580144782295bf6f36ded8f1415669.tix"
>
> Traceback (most recent call last):
> File "/Library/Python/2.5/site-packages/bzrlib/commands.py", line
> 729, in exception_to_return_code
> return the_callable(*args, **kwargs)
> File "/Library/Python/2.5/site-packages/bzrlib/commands.py", line
> 924, in run_bzr
> ret = run(*run_argv)
> File "/Library/Python/2.5/site-packages/bzrlib/commands.py", line
> 560, in run_argv_aliases
> return self.run(**all_cmd_args)
> File "/Library/Python/2.5/site-packages/bzrlib/builtins.py", line
> 2937, in run
> exclude=safe_relpath_files(tree, exclude))
> File "/Library/Python/2.5/site-packages/bzrlib/decorators.py", line
> 192, in write_locked
> result = unbound(self, *args, **kwargs)
> File "/Library/Python/2.5/site-packages/bzrlib/workingtree_4.py",
> line 226, in commit
> result = WorkingTree3.commit(self, message, revprops, *args,
> **kwargs)
> File "/Library/Python/2.5/site-packages/bzrlib/decorators.py", line
> 192, in write_locked
> result = unbound(self, *args, **kwargs)
> File "/Library/Python/2.5/site-packages/bzrlib/mutabletree.py",
> line 228, in commit
> *args, **kwargs)
> File "/Library/Python/2.5/site-packages/bzrlib/commit.py", line
> 397, in commit
> self.branch.repository, new_revno, self.rev_id)
> File "/Library/Python/2.5/site-packages/bzrlib/branch.py", line
> 821, in import_last_revision_info
> self.repository.fetch(source_repo, revision_id=revid)
> File "/Library/Python/2.5/site-packages/b...

Read more...

Martin Pool (mbp) wrote :

2009/8/14 dbmcclain <email address hidden>:
> Hello Martin,
>
> I would attach it for you, if I knew where it was and what it is
> named. I'm not a BZR expert, just a devoted developer, and I use BZR
> to keep safe copies of all my work and to distribute it to my other
> workstations.

Sorry, I thought the traceback would include the whole path but of
course it does not.

If you run 'bzr info' in the tree it'll tell you where the repository
directory is. If you look inside .bzr/repository/indices you should
find this file. It would also help if you can attach the pack-names
file.

These indexes will include some data about file names and the pattern
of commits but none of your actual source.

Do you recall bzr or your machine crashing or any filesystem problems
prior to this message?

--
Martin <http://launchpad.net/~mbp/>

Download full text (9.1 KiB)

Hello Martin,

I'll dig around for the file. The machines did not crash. But this is
the second time in 2 days that this message has happened. The BZR
repository lives on a FireWire connection to a Mac Mini. All machines
communicate via AirPort, or through a Ethernet cable to the AirPort
Controller device and then on to the Mac Mini.

The Mac Mini is running OS X Tiger, but from what I can tell, it has
no involvement with the BZR process except to furnish its disk as a
remote shared disk. The BZR seems to be running on each individual
development machine and treats that remote shared disk pretty much
like a local disk. I imagine that OS X is running some form of NFS??

The machine on which this error just happened is a late version iMac
running Leopard 10.5.8, 2.93 GHz, 4 GB Intel Core 2 Duo.

The repository has been running for almost 18 months, and the Lisp
project has about 1900 commits since Mar '08. It is a pretty large
project with the local BZR repository running around 3 GB? See ls
display...

   /Users/dbmcclain/.bzr/repository/packs:
   total used in directory 849464 available 581745268
   drwxr-xr-x 13 dbmcclain staff 442 Aug 13 23:48 .
   drwxr-xr-x 10 dbmcclain staff 340 Aug 13 23:48 ..
   -rw-r--r-- 1 dbmcclain staff 34857 Aug 13 23:47
1e1e729560110aa05ff86b78eb58eaf4.pack
   -rw-r--r-- 1 dbmcclain staff 3443 Aug 13 23:47
3979d885503543f660ab67b4f5bae6a8.pack
   -rw-r--r-- 1 dbmcclain staff 302221443 Jul 10 16:20
4a78a100b27c9acd5dea2e845b14bc46.pack
   -rw-r--r-- 1 dbmcclain staff 13202447 Aug 8 22:37
6fdffe51c8ccbb531519c279a3c4ae39.pack
   -rw-r--r-- 1 dbmcclain staff 30374 Aug 13 02:01
871fc3897819626f9f88ef9b748a3961.pack
   -rw-r--r-- 1 dbmcclain staff 61378 Aug 11 00:23
96f1f620050490fbb0ced23be9f3e94a.pack
   -rw-r--r-- 1 dbmcclain staff 47437 Aug 11 23:46
9cd5c68d5a65c65cc73af793e8f247b9.pack
   -rw-r--r-- 1 dbmcclain staff 5230463 Jul 27 15:02
b19d92247c23255ca30ebfb2a5813df9.pack
   -rw-r--r-- 1 dbmcclain staff 984 Aug 13 23:48
da89347a6c44bbea0e3c0d2b0334890d.pack
   -rw-r--r-- 1 dbmcclain staff 9452863 Jul 14 04:33
e6f69f414e56b946d22907f998a6a144.pack
   -rw-r--r-- 1 dbmcclain staff 104621833 Jul 10 16:05
f92c97146e1e854b7da2800534e51426.pack

Dr. David McClain
<email address hidden>

On Aug 14, 2009, at 02:03 AM, Martin Pool wrote:

> 2009/8/14 dbmcclain <email address hidden>:
>> Hello Martin,
>>
>> I would attach it for you, if I knew where it was and what it is
>> named. I'm not a BZR expert, just a devoted developer, and I use BZR
>> to keep safe copies of all my work and to distribute it to my other
>> workstations.
>
> Sorry, I thought the traceback would include the whole path but of
> course it does not.
>
> If you run 'bzr info' in the tree it'll tell you where the repository
> directory is. If you look inside .bzr/repository/indices you should
> find this file. It would also help if you can attach the pack-names
> file.
>
> These indexes will include some data about file names and the pattern
> of commits but none of your actual source.
>
> Do you ...

Read more...

Martin Pool (mbp) wrote :

Could you please run 'md5sums *' in the packs directory and check all
the packs are consistent with their filenames?

I'd like to get to the bottom of this but in the interim you might
find it more reliable to use sftp or ssh.

If that file has been truncated after bzr wrote it out I'm not sure if
we'll be able to recover that data from other files in the repository.
 The problem here is to do with reading back data that's already been
written, not with the commit itself. If you can easily go back to a
recent backup that'd probably be the best thing; you could also run
'bzr check' on a copy from the backup.

--
Martin <http://launchpad.net/~mbp/>

Download full text (7.0 KiB)

Hi Martin,

There seems to be no problem... I reran the BZR commit just after
getting the error message both times, and it appears to be okay. I
retrieved the lastest commits to other machines just fine.

I wonder if we are running up against size boundaries in the code. I
understand that BZR is written in Python, which means C or C++... Not
quite the capabilities I have with my Lisp systems...

- DM

On Aug 14, 2009, at 03:29 AM, Martin Pool wrote:

> Could you please run 'md5sums *' in the packs directory and check all
> the packs are consistent with their filenames?
>
> I'd like to get to the bottom of this but in the interim you might
> find it more reliable to use sftp or ssh.
>
> If that file has been truncated after bzr wrote it out I'm not sure if
> we'll be able to recover that data from other files in the repository.
> The problem here is to do with reading back data that's already been
> written, not with the commit itself. If you can easily go back to a
> recent backup that'd probably be the best thing; you could also run
> 'bzr check' on a copy from the backup.
>
> --
> Martin <http://launchpad.net/~mbp/>
>
> --
> ShortReadvError on tix file
> https://bugs.launchpad.net/bugs/413430
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Bazaar Version Control System: Incomplete
>
> Bug description:
> bash-3.2$ bzr commit -m "Mach compiler improvements, Bfly recv takes
> an optional argument for the received message"
> Committing to: /Volumes/Repository/srv/bzr/Lispworks/
> modified Butterfly/Source/Keep-Alive.lisp
> modified Butterfly/Source/Standard-Server.lisp
> modified Butterfly/Source/bfly-mailbox.lisp
> modified Butterfly/Source/bfly-message.lisp
> modified Butterfly/Source/bfly-socket2.lisp
> modified Butterfly/Source/bfly-spawn.lisp
> modified Butterfly/Source/fru-replace-code.lisp
> modified Okeanos/lock-daemon.lisp
> modified Okeanos/remote-access.lisp
> modified tools/useful-macros/match-macro-ex-opt.lisp
> bzr: ERROR: bzrlib.errors.ShortReadvError: readv() read 268 bytes
> rather than 4096 bytes at 171764 for
> "d0580144782295bf6f36ded8f1415669.tix"
>
> Traceback (most recent call last):
> File "/Library/Python/2.5/site-packages/bzrlib/commands.py", line
> 729, in exception_to_return_code
> return the_callable(*args, **kwargs)
> File "/Library/Python/2.5/site-packages/bzrlib/commands.py", line
> 924, in run_bzr
> ret = run(*run_argv)
> File "/Library/Python/2.5/site-packages/bzrlib/commands.py", line
> 560, in run_argv_aliases
> return self.run(**all_cmd_args)
> File "/Library/Python/2.5/site-packages/bzrlib/builtins.py", line
> 2937, in run
> exclude=safe_relpath_files(tree, exclude))
> File "/Library/Python/2.5/site-packages/bzrlib/decorators.py", line
> 192, in write_locked
> result = unbound(self, *args, **kwargs)
> File "/Library/Python/2.5/site-packages/bzrlib/workingtree_4.py",
> line 226, in commit
> result = WorkingTree3.commit(self, message, revprops, *args,
> **kwargs)
> File "/Library/Python/2.5/site-packages/bzrlib/decorators.py", line
> 192, in write_locked
> result = unbound(self, *args, **kwargs)
>...

Read more...

Martin Pool (mbp) wrote :

2009/8/14 dbmcclain <email address hidden>:
> Hi Martin,
>
> There seems to be no problem... I reran the BZR commit just after
> getting the error message both times, and it appears to be okay. I
> retrieved the lastest commits to other machines just fine.

I would recommend you run bzr check to make sure it's ok.

> I wonder if we are running up against size boundaries in the code. I
> understand that BZR is written in Python, which means C or C++... Not
> quite the capabilities I have with my Lisp systems...

I don't think that would cause us to see the file as truncated. It's
possible (unlikely) we'd have problems around the 2gb file size, but
it's unlikely the index file alone would be so big unless your
repository is truly huge. (Can you check?)

Based on this I would probably attribute it to an NFS or network glitch.

--
Martin <http://launchpad.net/~mbp/>

Download full text (7.4 KiB)

Hi Martin,

Thanks for the advice. You are teaching me some new tricks:

bash-3.2$ bzr check
Checking working tree at 'file:///Users/dbmcclain/projects/lispworks/'.
Checking repository at 'file:///Users/dbmcclain/'.
checked repository <bzrlib.transport.local.LocalTransport url=file:///Users/dbmcclain/
 > format <RepositoryFormatKnitPack1>
   2344 revisions
  17307 file-ids
  24783 unique file texts
4693654 repeated file texts
      0 unreferenced text versions
Checking branch at 'file:///Users/dbmcclain/projects/lispworks/'.
checked branch file:///Users/dbmcclain/projects/lispworks/ format
Branch format 6

On Aug 14, 2009, at 05:43 AM, Martin Pool wrote:

> 2009/8/14 dbmcclain <email address hidden>:
>> Hi Martin,
>>
>> There seems to be no problem... I reran the BZR commit just after
>> getting the error message both times, and it appears to be okay. I
>> retrieved the lastest commits to other machines just fine.
>
> I would recommend you run bzr check to make sure it's ok.
>
>> I wonder if we are running up against size boundaries in the code. I
>> understand that BZR is written in Python, which means C or C++... Not
>> quite the capabilities I have with my Lisp systems...
>
> I don't think that would cause us to see the file as truncated. It's
> possible (unlikely) we'd have problems around the 2gb file size, but
> it's unlikely the index file alone would be so big unless your
> repository is truly huge. (Can you check?)
>
> Based on this I would probably attribute it to an NFS or network
> glitch.
>
>
> --
> Martin <http://launchpad.net/~mbp/>
>
> --
> ShortReadvError on tix file
> https://bugs.launchpad.net/bugs/413430
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Bazaar Version Control System: Incomplete
>
> Bug description:
> bash-3.2$ bzr commit -m "Mach compiler improvements, Bfly recv takes
> an optional argument for the received message"
> Committing to: /Volumes/Repository/srv/bzr/Lispworks/
> modified Butterfly/Source/Keep-Alive.lisp
> modified Butterfly/Source/Standard-Server.lisp
> modified Butterfly/Source/bfly-mailbox.lisp
> modified Butterfly/Source/bfly-message.lisp
> modified Butterfly/Source/bfly-socket2.lisp
> modified Butterfly/Source/bfly-spawn.lisp
> modified Butterfly/Source/fru-replace-code.lisp
> modified Okeanos/lock-daemon.lisp
> modified Okeanos/remote-access.lisp
> modified tools/useful-macros/match-macro-ex-opt.lisp
> bzr: ERROR: bzrlib.errors.ShortReadvError: readv() read 268 bytes
> rather than 4096 bytes at 171764 for
> "d0580144782295bf6f36ded8f1415669.tix"
>
> Traceback (most recent call last):
> File "/Library/Python/2.5/site-packages/bzrlib/commands.py", line
> 729, in exception_to_return_code
> return the_callable(*args, **kwargs)
> File "/Library/Python/2.5/site-packages/bzrlib/commands.py", line
> 924, in run_bzr
> ret = run(*run_argv)
> File "/Library/Python/2.5/site-packages/bzrlib/commands.py", line
> 560, in run_argv_aliases
> return self.run(**all_cmd_args)
> File "/Library/Python/2.5/site-packages/bzrlib/builtins.py", line
> 2937, in run
> exclude=safe_relpath_files(tree, exclude))
> File "/Li...

Read more...

Martin Pool (mbp) wrote :

I also hit this. I would guess that the file corruption is due to an OS problem (a crash while writing perhaps), but there is a bug here that we don't give a good message, and that it's not very easy to see how to recover from it. This is distinct from bug 402857, which is about truncation of the pack files themselves.

summary: - ShortReadvError on tix file
+ ShortReadvError on index file
Changed in bzr:
importance: Medium → High
status: Incomplete → Confirmed
Martin Pool (mbp) on 2011-01-27
description: updated
Jelmer Vernooij (jelmer) on 2011-02-01
tags: added: data-integrity
tags: added: ui
Jelmer Vernooij (jelmer) wrote :

I just reproduced when I ran out of battery.

Jelmer Vernooij (jelmer) wrote :

FWIW, I haven't managed to reproduce this since your fdatasync patch landed.

Jens Berthold (jens-jebecs) wrote :

I also have this bug with etckeeper (automatic versioning of your /etc files) calling bzr.
Now when using "apt-get install ..." --> which calls "etckeeper post-install" --> which calls bzr, I get:
bzr: ERROR: bzrlib.errors.ShortReadvError: readv() read 0 bytes rather than 175 bytes at 0 for "6f3bd77ce343745312c8d43b11738469.rix"

Jelmer Vernooij (jelmer) on 2017-11-08
tags: added: check-for-breezy
Jelmer Vernooij (jelmer) on 2017-11-12
tags: removed: check-for-breezy
Changed in brz:
status: New → Triaged
importance: Undecided → High
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers