regression in snapshot performance in trusty

[Michael Hudson-Doyle]

'nova image-create' is a lot slower on a trusty openstack than a precise one:

+---------+--------+----------------+-------------------+
| ubuntu  | image  | disk size (GB) | snapshot time (s) |
+---------+--------+----------------+-------------------+
| precise | cirros | 1              | 6                 |
+---------+--------+----------------+-------------------+
| precise | cirros | 10             | 6                 |
+---------+--------+----------------+-------------------+
| precise | trusty | 10             | 38                |
+---------+--------+----------------+-------------------+
| trusty  | cirros | 1              | 15                |
+---------+--------+----------------+-------------------+
| trusty  | cirros | 10             | 61                |
+---------+--------+----------------+-------------------+
| trusty  | trusty | 10             | 100               |
+---------+--------+----------------+-------------------+

It's also interesting that the time in trusty appears to depend on the disk size whereas it didn't in trusty. AFAICT the nova code is the same in both havana and icehouse, leaving libvirt as the most likely culprit -- but this is not confirmed.

[Serge Hallyn (serge-hallyn)]

Thanks for submitting this bug report.

Is it possible to show the nova image create in terms of libvirt actions? Looking briefly at the nova source I didn't see anything that obviously was the equivalent of 'nova image create'. If you or Chuck cannot break it down here then I'll install and trace to figure it out.

[Michael Hudson-Doyle (mwhudson)]

Oh right yes, I actually went through the process of figuring this out, so it makes sense to share that rather than having you figure it out from scratch :-)

This script http://people.linaro.org/~mwhudson/snap.sh.txt does the same as nova-image create using the virsh binary (apart from the final upload to glance).

Using a --shallow clone and then flattening it is apparently to work around an old qemu bug:

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1638

and not doing that would, in my measurements on arm64 at least, help a bunch (unless I got very confused this code is present in havana too though so it can't be the cause of the regression).

[Serge Hallyn (serge-hallyn)]

Great - thanks for the script, I'll look into it

[Serge Hallyn (serge-hallyn)]

This seems to have gotten much better in utopic. Whereas in a trusty container I get

real 1m23.083s

in a utopic container I get

real 0m49.224s

[Ryan Harper (raharper)]

Using a slightly modified script (injecting sudo before each qemu-img command) here's my process for testing:

1. uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily release=trusty arch=amd64
2. uvt-kvm create --memory 1024 --cpu 1 --disk 10 trusty-test1 release=trusty arch=amd64
3. bash -x ./bench.sh trusty-test1

I ran this on up-to-date trusty and utopic on Intel NUCs with an SSD:

Trusty output (only the times):

real 0m0.063s (qemu-img create)
real 0m25.432s (virsh block copy)
real 0m4.774s (qemu-img convert)
real 0m32.475s (total)

Utopic output (only the times):

real 0m0.054s (qemu-img create)
real 0m25.862s (virsh block copy)
real 0m4.895s (qemu-img convert)
real 0m32.115s (total)

These numbers will go up as the VMs stay up and running, which may expose different performance characteristics.

Trusty Host:
(kriek) ~ % dpkg --list | egrep "(libvirt|qemu)"
ii ipxe-qemu 1.0.0+git-20131111.c3d1e78-2ubuntu1 all PXE boot firmware - ROM images for qemu
ii libvirt-bin 1.2.2-0ubuntu13.1 amd64 programs for the libvirt library
ii libvirt0 1.2.2-0ubuntu13.1 amd64 library for interfacing with different virtualization systems
ii python-libvirt 1.2.2-0ubuntu1 amd64 libvirt Python bindings
ii qemu-keymaps 2.0.0+dfsg-2ubuntu1 all QEMU keyboard maps
ii qemu-kvm 2.0.0+dfsg-2ubuntu1 amd64 QEMU Full virtualization on x86 hardware (transitional package)
ii qemu-system-common 2.0.0+dfsg-2ubuntu1 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-x86 2.0.0+dfsg-2ubuntu1 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 2.0.0+dfsg-2ubuntu1 amd64 QEMU utilities
ii uvtool-libvirt 0~bzr92-0ubuntu1 all Library and tools for using Ubuntu Cloud Images with libvirt

Utopic host:

(lambic) ~ % dpkg --list | egrep "(libvirt|qemu)"
ii ipxe-qemu 1.0.0+git-20131111.c3d1e78-2ubuntu1 all PXE boot firmware - ROM images for qemu
ii libvirt-bin 1.2.5-0ubuntu5 amd64 programs for the libvirt library
ii libvirt0 1.2.5-0ubuntu5 amd64 library for interfacing with different virtualization systems
ii python-libvirt 1.2.5-0ubuntu1 amd64 libvirt Python bindings
ii qemu-keymaps 2.0.0+dfsg-2ubuntu3 all QEMU keyboard maps
ii qemu-kvm 2.0.0+dfsg-2ubuntu3 amd64 QEMU Full virtualization on x86 hardware (transitional package)
ii qemu-system-common 2.0.0+dfsg-2ubuntu3 amd64 QEMU full system emulatio...

[Michael Hudson-Doyle (mwhudson)]

Ryan Harper <email address hidden> writes:

> Using a slightly modified script (injecting sudo before each qemu-img
> command) here's my process for testing:
>
> 1. uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily release=trusty arch=amd64
> 2. uvt-kvm create --memory 1024 --cpu 1 --disk 10 trusty-test1 release=trusty arch=amd64
> 3. bash -x ./bench.sh trusty-test1
>
> I ran this on up-to-date trusty and utopic on Intel NUCs with an SSD:

> Trusty output (only the times):
>
> real 0m0.063s (qemu-img create)
> real 0m25.432s (virsh block copy)
> real 0m4.774s (qemu-img convert)
> real 0m32.475s (total)
>
> Utopic output (only the times):
>
> real 0m0.054s (qemu-img create)
> real 0m25.862s (virsh block copy)
> real 0m4.895s (qemu-img convert)
> real 0m32.115s (total)

Interesting. How much RAM do you have on the host? I think one of the
things that makes quite a large difference is if the image file created
by virsh block copy stays in the file cache. If it does, the qemu-img
convert is very fast. If it does not, it takes 10s of seconds... so
maybe this is all I've been seeing and the other changes are irrelevant
:/

Cheers,
mwh

> These numbers will go up as the VMs stay up and running, which may
> expose different performance characteristics.
>
> Trusty Host:
> (kriek) ~ % dpkg --list | egrep "(libvirt|qemu)"
> ii ipxe-qemu 1.0.0+git-20131111.c3d1e78-2ubuntu1 all PXE boot firmware - ROM images for qemu
> ii libvirt-bin 1.2.2-0ubuntu13.1 amd64 programs for the libvirt library
> ii libvirt0 1.2.2-0ubuntu13.1 amd64 library for interfacing with different virtualization systems
> ii python-libvirt 1.2.2-0ubuntu1 amd64 libvirt Python bindings
> ii qemu-keymaps 2.0.0+dfsg-2ubuntu1 all QEMU keyboard maps
> ii qemu-kvm 2.0.0+dfsg-2ubuntu1 amd64 QEMU Full virtualization on x86 hardware (transitional package)
> ii qemu-system-common 2.0.0+dfsg-2ubuntu1 amd64 QEMU full system emulation binaries (common files)
> ii qemu-system-x86 2.0.0+dfsg-2ubuntu1 amd64 QEMU full system emulation binaries (x86)
> ii qemu-utils 2.0.0+dfsg-2ubuntu1 amd64 QEMU utilities
> ii uvtool-libvirt 0~bzr92-0ubuntu1 all Library and tools for using Ubuntu Cloud Images with libvirt
>
> Utopic host:
>
> (lambic) ~ % dpkg --list | egrep "(libvirt|qemu)"
> ii ipxe-qemu 1.0.0+git-20131111.c3d1e78-2ubuntu1 all PXE boot firmware - ROM images for qemu
> ii libvirt-bin 1.2.5-0ubuntu5 amd64 programs for the libvirt library
> ii libvirt0 1.2.5-0ubuntu5 amd64 library for interfacing with different virtualizatio...

[Serge Hallyn (serge-hallyn)]

Is this still reproducible? (If so, in precisely what environment?)

[Michael Hudson-Doyle (mwhudson)]

Serge Hallyn <email address hidden> writes:

> Is this still reproducible? (If so, in precisely what environment?)

I am no longer working on this sort of thing. My most recent theory was
that there is no real regression, but there is a strong dependence on
available system RAM as to what is the fastest way to snapshot (but I
can't remember what the two ways are :-)

Cheers,
mwh

[Serge Hallyn (serge-hallyn)]

Thanks for the update.