I/O errors seen on tmpfs

[Clint Byrum]

Occasionally when using diskimage-builder on a large system with a large TMPFS, we see this error:

qemu-img: error while compressing sector 2069504: Input/output error

The exact operation that is happening is that qemu-img is taking a .raw file that is on a tmpfs partition and converting it to a qcow2 image. However, sometimes we also see errors when we create a loopback device and mkfs onto it.

The exact steps to reproduce:

git clone https://github.com/stackforge/diskimage-builder.git
cd diskimage-builder
bin/disk-image-create -x -a i386 base

It does not happen every time, maybe 1 in 10. However I have seen it on both Ubuntu 12.10 systems and Ubuntu 13.04, all on amd64. Also many times the errors are accompanied by these:

May 13 16:09:57 clint-HP kernel: [975578.653245] Buffer I/O error on device loop2p1, logical block 1943681
May 13 16:09:57 clint-HP kernel: [975578.653251] Buffer I/O error on device loop2p1, logical block 1943682
May 13 16:09:57 clint-HP kernel: [975578.653256] Buffer I/O error on device loop2p1, logical block 1943683
May 13 16:09:57 clint-HP kernel: [975578.653261] Buffer I/O error on device loop2p1, logical block 1943684
May 13 16:09:57 clint-HP kernel: [975578.653266] Buffer I/O error on device loop2p1, logical block 1943685
May 13 16:09:57 clint-HP kernel: [975578.653271] Buffer I/O error on device loop2p1, logical block 1943686
May 13 16:09:57 clint-HP kernel: [975578.653275] Buffer I/O error on device loop2p1, logical block 1943687
May 13 16:09:57 clint-HP kernel: [975578.653283] Buffer I/O error on device loop2p1, logical block 1943680
May 13 16:09:57 clint-HP kernel: [975578.653288] Buffer I/O error on device loop2p1, logical block 1943681

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: linux-image-3.8.0-19-generic 3.8.0-19.30
ProcVersionSignature: Ubuntu 3.8.0-19.29-generic 3.8.8
Uname: Linux 3.8.0-19-generic x86_64
ApportVersion: 2.9.2-0ubuntu8
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: clint 3245 F.... pulseaudio
 /dev/snd/controlC0: clint 3245 F.... pulseaudio
Date: Tue May 14 10:05:58 2013
HibernationDevice: RESUME=UUID=b753173e-b967-403e-b076-b1cfb5bc6e80
InstallationDate: Installed on 2013-01-14 (120 days ago)
InstallationMedia: Ubuntu 12.10 "Quantal Quetzal" - Release amd64+mac (20121017.3)
MachineType: Hewlett-Packard HP EliteBook Folio 9470m
MarkForUpload: True
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.8.0-19-generic root=/dev/mapper/ubuntu-root ro quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.8.0-19-generic N/A
 linux-backports-modules-3.8.0-19-generic N/A
 linux-firmware 1.106
SourcePackage: linux
UpgradeStatus: Upgraded to raring on 2013-04-30 (13 days ago)
dmi.bios.date: 09/24/2012
dmi.bios.vendor: Hewlett-Packard
dmi.bios.version: 68IBD Ver. F.30
dmi.board.name: 18DF
dmi.board.vendor: Hewlett-Packard
dmi.board.version: KBC Version 62.13
dmi.chassis.asset.tag: CNU2439PZT
dmi.chassis.type: 10
dmi.chassis.vendor: Hewlett-Packard
dmi.modalias: dmi:bvnHewlett-Packard:bvr68IBDVer.F.30:bd09/24/2012:svnHewlett-Packard:pnHPEliteBookFolio9470m:pvrA1029D1103:rvnHewlett-Packard:rn18DF:rvrKBCVersion62.13:cvnHewlett-Packard:ct10:cvr:
dmi.product.name: HP EliteBook Folio 9470m
dmi.product.version: A1029D1103
dmi.sys.vendor: Hewlett-Packard

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Can you see if this can also be reproduced on the latest mainline kernel, version 3.10-rc1:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.10-rc1-saucy/

Also, to reproduce this, can you provide some additional details on how large a system is required? Does it require a minimum amount of RAM, num cpus, etc? Does TMPFS need to be a minimum size?

Thanks in advance

[Clint Byrum (clint-fewbar)]

Excerpts from Joseph Salisbury's message of 2013-05-14 18:46:30 UTC:
> Can you see if this can also be reproduced on the latest mainline kernel, version 3.10-rc1:
> http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.10-rc1-saucy/
>
> Also, to reproduce this, can you provide some additional details on how
> large a system is required? Does it require a minimum amount of RAM,
> num cpus, etc? Does TMPFS need to be a minimum size?
>

I have seen this on a system with 16G of RAM, and one with 96G of RAM.
Any less than 4G of RAM will result in diskimage-builder not using tmpfs.

Will try mainline kernel ASAP.

[Phillip Susi (psusi)]

What is the size limit on the tmpfs and how are you creating the image? My guess is that you create a sparse image that would exceed the size limit if it were not sparse, then when you try to write to it, you end up hitting the size limit.

[Robert Collins (lifeless)]

http://lists.gnu.org/archive/html/qemu-devel/2013-04/msg02662.html