Running karmic as virtual machine with virtio hard disk outputs I/O errors

Bug #420423 reported by Nikolai Bochev on 2009-08-28
This bug affects 3 people
Affects Status Importance Assigned to Milestone
linux (Fedora)
Fix Released
linux (Ubuntu)
Andy Whitcroft
qemu-kvm (Ubuntu)
Dustin Kirkland 

Bug Description

When i try to run Karmic Alpha 4 on a Jaunty machine with KVM, i get the following in the log files all the time :

Aug 28 12:30:30 karmic kernel: [ 296.502337] end_request: I/O error, dev vda, sector 0

This happens only if the hard disk of the virtual machine is with type Virtio ( i.e. /dev/vda ). If i use "IDE DISK" as a device type everything runs flawlessly.

Created attachment 355795
dumpxml of the rawhide guest from virsh

Opening as a rawhide kernel bug, but it could also be a problem with the host kernel or qemu instance.

I have a rawhide kvm guest running on a Fedora 10 host. The guest's ring buffer is continually being spammed several times per second with this message:

end_request: I/O error, dev vda, sector 0

...everything seems to be working OK otherwise.

The guest is currently running:


...but the problem has existed for quite some time with earlier kernels too. The host is running:

...and has:


The guest is using the virtio block driver for the disk. Let me know if any other info would be useful.

Since we have a report of this on an F-11 host too, it sounds like a guest kernel bug:

But AFAICS, we should only get this if:

  1) There's is a read/write error on the disk image in the host. Of note
     here is that the disk image is an LVM volume /dev/rootvg64/rawhide

  2) The guest issues a SCSI command

Perhaps it's the serial number support:;a=commitdiff;h=1d589bb1

or the SG_IO passthru support:;a=commitdiff;h=1cde26f9

Both were added in 2.6.31-rc1

Mark asked me to try kernel-2.6.30-1.fc12.x86_64. With that kernel, I do not see the error message.

okay, the error is coming from here:

bool blk_do_ordered(struct request_queue *q, struct request **rqp)
        struct request *rq = *rqp;
        const int is_barrier = blk_fs_request(rq) && blk_barrier_rq(rq);

        if (!q->ordseq) {
                if (!is_barrier)
                        return true;

                if (q->next_ordered != QUEUE_ORDERED_NONE)
                        return start_ordered(q, rqp);
                else {
                         * Queue ordering not supported. Terminate
                         * with prejudice.
                        __blk_end_request_all(rq, -EOPNOTSUPP);

partial stack trace:

 [<ffffffff81254768>] blk_update_request+0xca/0x363
 [<ffffffff8127bc00>] ? debug_object_deactivate+0x47/0xf2
 [<ffffffff81254a30>] blk_update_bidi_request+0x2f/0x7f
 [<ffffffff812565b2>] __blk_end_request_all+0x44/0x74
 [<ffffffff8125c096>] blk_do_ordered+0x1e0/0x2ae
 [<ffffffff81256756>] blk_peek_request+0x174/0x1c8
 [<ffffffffa00102d3>] do_virtblk_request+0x192/0x1d3 [virtio_blk]
 [<ffffffff8125710f>] __blk_run_queue+0x54/0x9a

Okay, reverting this:;a=commitdiff;h=52b1fd5a27

  commit 52b1fd5a27c625c78373e024bf570af3c9d44a79
  Author: Mikulas Patocka <email address hidden>
  Date: Mon Jun 22 10:12:21 2009 +0100

    dm: send empty barriers to targets in dm_flush

    Pass empty barrier flushes to the targets in dm_flush().

    Signed-off-by: Mikulas Patocka <email address hidden>
    Signed-off-by: Alasdair G Kergon <email address hidden>

fixes it for me

Also, this little hack fixes it too:

  @@ -1163,7 +1163,7 @@ static int __make_request(struct request_queue *q, struct
          const int unplug = bio_unplug(bio);
          int rw_flags;

  - if (bio_barrier(bio) && bio_has_data(bio) &&
  + if (bio_barrier(bio) && /* bio_has_data(bio) && */
              (q->next_ordered == QUEUE_ORDERED_NONE)) {
                  bio_endio(bio, -EOPNOTSUPP);
                  return 0;

virtio_blk doesn't support barriers, and it seems these empty barriers submitted by device-mapper are getting through to the device and causing these errors

okay, sent patch upstream:

Created attachment 356487

Empty barriers are the mechanism now used to request block device flushes.

After the block layer changed, we had no flushing support through dm devices for a long time as the new method was tricky for us to use. Eventually we worked around the problems and the solution is mostly in place, as you have discovered.

You should think about whether you're limiting the applicability/usefulness of your software if you don't also implement support for empty barriers in virtio_blk.

Agreed. I think virtio_blk needs to support barriers if it's going to be at all useful. Otherwise, VM's that use it would be subject to data integrity problems in the face of crashes, right?

can we continue the discussion on lkml about whether barrier support should be required?

response from hch was:

  virtio_blk on kvm does not support any barriers at all, similar to many
  other drivers out there. If the queue flags say we don't support
  barriers higher layers should not submit any barrier requests.

wrt. immediate F-12 concerns, we should focus on getting rid of these errors

The patch has been applied to the F12 kernel to address the immediate concerns.

I think the patch makes sense.

Okay, patch applied in rawhide:

* Thu Aug 06 2009 Justin M. Forbes <email address hidden> 2.6.31-0.138.rc5.git3
- Fix kvm virtio_blk errors (#514901)

Not yet tagged for f12

The kernel patch and discussion for this issue is here:

Dave Hall (skwashd) wrote :

This looks like a dupe of #432959

Andy Whitcroft (apw) wrote :

Reviewing that patch thread there was no objection to it, but it has not yet hit mainline. Waiting for something more concrete here.

tags: added: regression-potential
Changed in ubuntu:
milestone: none → ubuntu-9.10
affects: ubuntu → linux (Ubuntu)
Changed in linux (Ubuntu):
importance: Undecided → Medium
status: New → Triaged
Andy Whitcroft (apw) on 2009-10-09
Changed in linux (Ubuntu):
assignee: nobody → Andy Whitcroft (apw)
status: Triaged → In Progress
Andy Whitcroft (apw) wrote :

I have forward ported the proposed fix from patchworks in comment #1 and build some kernels for testing. If those of you affected could give these a test and report back here I would be grateful. Note that as this affects barrier code there is some risk, do not use on critical data. Kernels can be found at the URL below:

Dave Hall (skwashd) wrote :

Hi Andy,

Thanks for working on this bug. I have tested the AMD64 kernel under KVM and haven't seen the error messages.

I am not running any production VMs using karmic at this stage, so simulating real load wasn't possible. I did run dd with it dumping 25G to disk and so saw no errors. Before loading the test kernel the errors were frequent every time there disk io.



Nikolai Bochev (n-bochev) wrote :

I will test as soon as possible and see if this kernel fixes things in my case.

Changed in linux (Ubuntu):
importance: Medium → High
Changed in linux (Fedora):
status: Unknown → Fix Released
Changed in qemu-kvm (Ubuntu):
status: New → In Progress
assignee: nobody → Dustin Kirkland (kirkland)
importance: Undecided → High
Dustin Kirkland  (kirkland) wrote :

There's a related bug in qemu-kvm, which I'm about to upload a fix for, affecting virtio and qcow2 disks.

Andy's kernel fix is needed too. I'm testing that now.


summary: Running karmic as virtual machine with virtio hard disk outputs I/O
- erros
+ errors
Dustin Kirkland  (kirkland) wrote :


Brilliant! Works like a champ!

Can we please get this into Karmic ASAP!


Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu-kvm - 0.11.0-0ubuntu2

qemu-kvm (0.11.0-0ubuntu2) karmic; urgency=low

  * debian/patches/09_qemu-fix-qcow2-backing-file-with-virtio.patch:
    cherry-pick patch from Fedora, fix virtio/qcow2 issues, LP: #420423
  * debian/patches/10_qemu-allow-pulseaudio-to-be-the-default.patch:
    needed to complete the fix for LP: #304649
  * debian/control: improve description of qemu-kvm-extras, LP: #448655

 -- Dustin Kirkland <email address hidden> Sun, 11 Oct 2009 13:16:11 -0500

Changed in qemu-kvm (Ubuntu):
status: In Progress → Fix Released
Andy Whitcroft (apw) wrote :

Proposing for inclusion in the karmic release kernel.

Andy Whitcroft (apw) on 2009-10-13
Changed in linux (Ubuntu):
status: In Progress → Fix Committed
Dave Hall (skwashd) wrote :

Andy thanks for the quick response on this one. Far better than some of the "enterprise" distros are at fixing bug.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 2.6.31-14.46

linux (2.6.31-14.46) karmic; urgency=low

  [ Andy Whitcroft ]

  * reinstate with no flavours
    - LP: #449637
  * [Upstream] elevator: fix fastfail checks to allow merge of readahead
    - LP: #444915
  * [Upstream] block: silently error unsupported empty barriers too
    - LP: #420423

  [ John Johansen ]

  * SAUCE: AppArmor: Fix mediation of "deleted" paths
    - LP: #415632

  [ Tim Gardner ]

  * [Config] CONFIG_X86_MCE
  * Revert "appletalk: Fix skb leak when ipddp interface is not loaded,
    CVE-2009-2903" - Use patch from which is slightly different.

  [ Upstream Kernel Changes ]

  * x86: fix csum_ipv6_magic asm memory clobber
  * tty: Avoid dropping ldisc_mutex over hangup tty re-initialization
  * x86: Don't leak 64-bit kernel register values to 32-bit processes
  * tracing: correct module boundaries for ftrace_release
  * ftrace: check for failure for all conversions
  * futex: fix requeue_pi key imbalance
  * futex: Move exit_pi_state() call to release_mm()
  * futex: Nullify robust lists after cleanup
  * futex: Fix locking imbalance
  * NOHZ: update idle state also when NOHZ is inactive
  * ima: ecryptfs fix imbalance message
  * libata: fix incorrect link online check during probe
  * sound: via82xx: move DXS volume controls to PCM interface
  * ASoC: WM8350 capture PGA mutes are inverted
  * KVM: Prevent overflow in KVM_GET_SUPPORTED_CPUID
  * KVM: VMX: flush TLB with INVEPT on cpu migration
  * KVM: fix LAPIC timer period overflow
  * KVM: SVM: Fix tsc offset adjustment when running nested
  * KVM: SVM: Handle tsc in svm_get_msr/svm_set_msr correctly
  * net: Fix wrong sizeof
  * mm: add_to_swap_cache() must not sleep
  * sis5513: fix PIO setup for ATAPI devices
  * PIT fixes to unbreak suspend/resume (bug #14222)
  * IMA: open new file for read
  * ACPI: Clarify resource conflict message
  * ACPI: fix Compaq Evo N800c (Pentium 4m) boot hang regression
  * net: restore tx timestamping for accelerated vlans
  * net: unix: fix sending fds in multiple buffers
  * tun: Return -EINVAL if neither IFF_TUN nor IFF_TAP is set.
  * net: Fix sock_wfree() race
  * smsc95xx: fix transmission where ZLP is expected
  * sky2: Set SKY2_HW_RAM_BUFFER in sky2_init
  * appletalk: Fix skb leak when ipddp interface is not loaded
  * ax25: Fix possible oops in ax25_make_new
  * ax25: Fix SIOCAX25GETINFO ioctl
  * sit: fix off-by-one in ipip6_tunnel_get_prl
  * Linux
  * drm/i915: Fix FDI M/N setting according with correct color depth
    - LP: #416792

 -- Andy Whitcroft <email address hidden> Tue, 13 Oct 2009 12:06:59 +0100

Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
tags: added: iso-testing
Changed in linux (Fedora):
importance: Unknown → Medium
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.