9p virtual file system on qemu slow

[max]

Hi,
The 9p virtual file system is slow. Several examples below:
---------------------------------------------------------
Host for the first time:
$ time ls bam.unsorted/
...........................
real 0m0.084s
user 0m0.000s
sys 0m0.028s
--------------------------------------------------
Host second and following:

real 0m0.009s
user 0m0.000s
sys 0m0.008s
------------------------------------------------------
VM for the first time:
$ time ls bam.unsorted/
................................
real 0m23.141s
user 0m0.064s
sys 0m2.156s
------------------------------------------------------
VM for the second time
real 0m3.643s
user 0m0.024s
sys 0m0.424s
----------------------------------------------------

Copy on host:
$ time cp 5173T.root.bak test.tmp
real 0m30.346s
user 0m0.004s
sys 0m5.324s

$ ls -lahs test.tmp
2.7G -rw------- 1 oneadmin cloud 2.7G Mar 26 21:47 test.tmp

---------------------------------------------------
$ copy on VM for the same file

time cp 5173T.root.bak test.tmp

real 5m46.978s
user 0m0.352s
sys 1m38.962s

[Serge Hallyn (serge-hallyn)]

Thanks for taking the time to report this bug. Which release are you currently using? Is this performance a regression over past releases? Would it be possible for you to test with upstream qemu to see whether performance has improved?

[max (mikhmv)]

Hi Serge,

Here are info:

max@s0:/var/lib/one/var$ uname -a
Linux s0 3.2.0-20-generic #33-Ubuntu SMP Tue Mar 27 16:42:26 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

max@s0:/var/lib/one/var$ kvm --version
QEMU emulator version 1.0 (qemu-kvm-devel), Copyright (c) 2003-2008 Fabrice Bellard

 ps aux| grep kvm | grep one-52
oneadmin 62687 28.9 0.3 25283292 839204 ? Sl 01:31 2:22 /usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 20040 -smp 60,sockets=60,cores=1,threads=1 -name one-52 -uuid 30c77a47-85c4-65dd-6c21-9d9b3e66cabe -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-52.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -drive file=/var/lib/one/var//52/images/disk.0,if=none,id=drive-virtio-disk0,format=raw,cache=writeback -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/one/var//52/images/disk.1,if=none,id=drive-virtio-disk4,format=raw,cache=writeback -device virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk4,id=virtio-disk4 -fsdev local,security_model=mapped,id=fsdev-fs0,path=/tank/biouml-shared -device virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=VirtFS,bus=pci.0,addr=0x3 -netdev tap,fd=21,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:00:c0:a8:02:1e,bus=pci.0,addr=0x4 -usb -device usb-mouse,id=input0 -vnc 0.0.0.0:52 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -cpu host

[Serge Hallyn (serge-hallyn)]

Thanks.

I'm posting a new package based on today's git head of upstream at ppa:ubuntu-virt/ppa. When (and if) that builds, is it possible for you to try with that version? Then we will know whether to file this bug upstream, or find and cherrypick the fixes.

[max (mikhmv)]

Hi Serge,

syslogs on host completely clean for all my 3 bugs. No messages at all regarding any errors.

I just added this repository. I can basically update now daily as I have to reboot server daily to make kvm working.
I installed qemu version: 1.0+noroms-20120220-0ubuntu1.

[Serge Hallyn (serge-hallyn)]

Quoting max (<email address hidden>):
> Hi Serge,
>
> syslogs on host completely clean for all my 3 bugs. No messages at all
> regarding any errors.

Thanks.

> I just added this repository. I can basically update now daily as I have to reboot server daily to make kvm working.
> I installed qemu version: 1.0+noroms-20120220-0ubuntu1.

1.0+noroms-20120330-0ubuntu1 should now be available, though of course
if the issue is fixed in 1.0+noroms-20120220-0ubuntu1 then that's even
more helpful :)

[max (mikhmv)]

Hi Serge,

I tried new build. It is not working for me. I got this message:

max@s0:/var/lib/one/var$ virsh create 52/deployment.0
error: Failed to create domain from 52/deployment.0
error: internal error process exited while connecting to monitor: kvm: -fsdev local,security_model=mapped,id=fsdev-fs0,path=/tank/biouml-shared: there is no option group "fsdev"
fsdev is not supported by this qemu build.

[Serge Hallyn (serge-hallyn)]

Drat.

I'll simply need to try to reproduce.

[max (mikhmv)]

Hi Serge,

qemu build 0330 does not support -fsdev. I was not able to start kvm with 9p at all.

If you will make another build I will try it too.

Thanks

[Serge Hallyn (serge-hallyn)]

Quoting max (<email address hidden>):
> Hi Serge,
>
> qemu build 0330 does not support -fsdev. I was not able to start kvm
> with 9p at all.
>
> If you will make another build I will try it too.

Thanks. Sorry about that. I'm hoping I just used an old packaging
tree as my starting base, will re-try, and post here when it has
built.

[Serge Hallyn (serge-hallyn)]

Hi,

version 1.0+noroms-20120330-0ubuntu3 has been built. Could you verify whether that fixes the issue (as well as your others)?

[max (mikhmv)]

Thanks Serge,

I installed it and now testing. Let's wait for several days. I will write if have any issues.

[max (mikhmv)]

Hi Serge,
Stability bugs was fixed in KVM, thanks!
Unfortunately this one still here:

----------------------------------------------------------------------------
VM:

 ls -las XXXXX.bam
14537970 -rw-r--r-- 1 10001 cloud 14885246140 Mar 23 12:19 XXXXX.bam

time cp XXXX.bam test.tmp

real 14m45.580s
user 0m0.420s
sys 1m45.823s

~16 Mb/sec
--------------------------------------------------------------
Host:
$ ls -las 5173N_sorted_dedup_rg_dd2_kar.chr4.ra.dd.recal.bam
15220278 -rw-r--r-- 1 oneadmin cloud 15583853314 Mar 27 18:53 YYYY.bam

time cp YYYYY.bam test.tmp

real 4m38.525s
user 0m0.048s
sys 1m11.124s

~53MB/sec

[Serge Hallyn (serge-hallyn)]

Thanks, Max. Marked as affecting upstream QEMU per the last comment.

[Iggy (iggy-theiggy)]

Can you try with security_model=passthrough?

[max (mikhmv)]

One of possible problems could be a block size. In this case I am using ZFS with raidZ 4+1 drives. Each drive has 4Kb block. So optimal block size is 16384 bytes. By optimizing block size it possible to improve performance 10 folds but 9p stably provides 10 folds worse performance than native writes.

Some extra tests:
---------------------
VM - mapped

$ dd if=/dev/zero of=test count=100000
100000+0 records in
100000+0 records out
51200000 bytes (51 MB) copied, 20.7879 s, 2.5 MB/s

$ dd if=/dev/zero of=test count=100000 bs=16384
100000+0 records in
100000+0 records out
1638400000 bytes (1.6 GB) copied, 74.4378 s, 22.0 MB/s
------------------------------------------------------------
Host:
$ dd if=/dev/zero of=test count=100000
100000+0 records in
100000+0 records out
51200000 bytes (51 MB) copied, 1.60118 s, 32.0 MB/s

$ dd if=/dev/zero of=test count=100000 bs=16384
100000+0 records in
100000+0 records out
1638400000 bytes (1.6 GB) copied, 4.89932 s, 334 MB/s

Iggy: I has issue with permission in passthrough mode. Can you give an idea how to setup permissions in this mode?

[max (mikhmv)]

>>>Can you try with security_model=passthrough?
It provides the same results, see below:

$ dd if=/dev/zero of=test count=100000
100000+0 records in
100000+0 records out
51200000 bytes (51 MB) copied, 19.8581 s, 2.6 MB/s

$ dd if=/dev/zero of=test count=100000 bs=16384
100000+0 records in
100000+0 records out
1638400000 bytes (1.6 GB) copied, 72.3009 s, 22.7 MB/s

[M. Mohan Kumar (mohan-6)]

Hi Max,

Could you try passing msize=262144 for 9p mount point and post the results?

Host:
[root@llm116 media]# ls -lhas file
1.1G -rw-r--r-- 1 root root 1.0G Apr 26 11:05 file

[root@llm116 media]# dd if=/dev/zero of=file bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.700828 s, 1.5 GB/s

[root@llm116 media]# time cp file file2

real 0m6.353s
user 0m0.007s
sys 0m1.520s

VM:

[root@qemu-img-64 pass]# time cp file 9p_file

real 0m12.261s
user 0m0.154s
sys 0m2.582s

[root@qemu-img-64 pass]# dd if=/dev/zero of=file.9 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.07335 s, 518 MB/s

[root@qemu-img-64 pass]# mount
[snip]
v_pass on /pass type 9p (rw,trans=virtio,version=9p2000.L,msize=262144)
[/snip]

[max (mikhmv)]

Hi Mohan,
this parameter provide significant improvement in big file access/write:
--------------------------------------------------------------------------------------------
VirtFS on /srv/shared type 9p (rw,trans=virtio,version=9p2000.L,msize=262144)

$ dd if=/dev/zero of=test count=100000 bs=16384
100000+0 records in
100000+0 records out
1638400000 bytes (1.6 GB) copied, 36.3589 s, 45.1 MB/s

l$ dd if=/dev/zero of=test count=100000
100000+0 records in
100000+0 records out
51200000 bytes (51 MB) copied, 25.6597 s, 2.0 MB/s

$ dd if=/dev/zero of=test count=1024 bs=262144
1024+0 records in
1024+0 records out
268435456 bytes (268 MB) copied, 3.41936 s, 78.5 MB/s

Speed of copy for large file ~45MB/s (read and write from the same disk).
---------------------------------------------------------------------------------------------
But Host:
time ls -lahs bam.unsorted/
many files:
real 0m0.053s
user 0m0.004s
sys 0m0.036s

VM:
real 0m4.449s
user 0m0.012s
sys 0m0.136s
--------------------------------------------------------
So we have delays on the first file access.
Is it possible to resolve this issue?

[Thomas Huth (th-huth)]

Can you still reproduce this problem with the latest version of QEMU (currently version 2.9.0)?

[Launchpad Janitor (janitor)]

[Expired for QEMU because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for qemu-kvm (Ubuntu) because there has been no activity for 60 days.]