qemu-nbd slow and missing "writeback" cache option
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
QEMU |
Invalid
|
Undecided
|
Unassigned | ||
qemu-kvm (Ubuntu) |
Expired
|
Undecided
|
Unassigned |
Bug Description
Binary package hint: qemu-kvm
dpkg -l | grep qemu
ii kvm 1:84+dfsg-
ii qemu 0.12.3+
ii qemu-common 0.12.3+
ii qemu-kvm 0.12.3+
ii qemu-kvm-extras 0.12.3+
ii qemu-launcher 1.7.4-1ubuntu2 GTK+ front-end to QEMU computer emulator
ii qemuctl 0.2-2 controlling GUI for qemu
lucid amd64.
qemu-nbd is a lot slower when writing to disk than say nbd-server.
It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.
The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).
The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.
Attached is a patch that adds a --cache=
--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.
Consider this test scenario:
dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0
With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).
With cache=off, it takes about 30 seconds.
With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server
Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.
Changed in qemu-kvm (Ubuntu): | |
status: | New → Confirmed |
tags: | added: patch |
Changed in qemu-kvm (Ubuntu): | |
status: | Incomplete → Confirmed |
Changed in qemu-kvm (Ubuntu): | |
status: | Confirmed → Incomplete |
Noone has confirmed, but have passed along to upstream. If upstream
takes this patch then we will likely pull it into our patchset.