Changing use_cow_images breaks existing instances

[Josias Montag]

Summary:
Changing use_cow_images from false to true breaks all existing instances with raw images.

Env:
Nova version 3.3.1, Fuel 9.0.

Reproduce steps:
1. Set use_cow_images=false in nova.conf
2. Create and start an instance using a raw image
3. Set use_cow_images=true in nova.conf
4. Restart the instance

Expect result:
The existing instance restarts and continues to work.

Actual result:
Restarting the instances fails with "Image is not in qcow2 format". Full log:

oslo_messaging.rpc.dispatcher [req-23c83b76-ee13-44e7-9b92-c7ac20f369cc 91c65783f69e4322992d4b7f86172fab 19e9d9a752d14ec8a20a0d8be1681bdd - - -] Exception during message handling: internal error: process exited while connecting to monitor: 2016-09-23T11:56:59.855117Z qemu-system-x86_64: -drive file=/var/lib/nova/instances/86682c9d-628a-4ab6-9836-1ab3bc5a493b/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: Image is not in qcow2 format
2016-09-23 11:57:01.145 30847 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-09-23 11:57:01.145 30847 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-09-23 11:57:01.145 30847 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-09-23 11:57:01.145 30847 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-09-23 11:57:01.145 30847 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-09-23 11:57:01.145 30847 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-09-23 11:57:01.145 30847 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-09-23 11:57:01.145 30847 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 110, in wrapped
2016-09-23 11:57:01.145 30847 ERROR oslo_messaging.rpc.dispatcher payload)
2016-09-23 11:57:01.145 30847 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-23 11:57:01.145 30847 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-09-23 11:57:01.145 30847 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/ex[....]

[Matt Riedemann (mriedem)]

What release is nova in fuel 9.0? Liberty? Mitaka? I don't know what 3.3.1 maps to.

[Josias Montag (monday)]

Sorry, my mentioned version seems to be the version of the client.
It is OpenStack Mitaka.

~# nova-compute --version
13.0.0

[Matthew Booth (mbooth-9)]

This is a known limitation.

The issue is that the required information: what format an existing disk is in, isn't stored anywhere independently of the host configuration. That is, we only know what format the disk is in from the configuration of the compute host. If you change that, you're changing our store of what the disk format is, and it's going to break.

Incidentally changing this on a running compute exposes a severe security vulnerability:

Admin sets use_cow_images = False
User creates instance with an ephemeral disk, created as raw.
User overwrites beginning of ephemeral disk with a fake qcow2 header with backing file '/dev/sda'.

Admin sets use_cow_images = True
User reboots their instance.
Nova tells libvirt disk is qcow2 (from config), qemu finds valid qcow2 header, starts guest.
User logs in to their instance, ephemeral disk is now the contents of /dev/sda from the compute host.

IoW you absolutely MUST NOT do this.

The persistent libvirt metadata spec is intended to address this, as it stores disk metadata independently of the host configuration. That may land during Ocata, but it will never be backportable to Mitaka.

There is a workaround, btw. You can't change this on a host (ever), and having a heterogeneous environment is also bad, although we have a workaround for cold-migration between hosts with different config which at least eliminates the security bug. However, I believe you should be able to live (NB *NOT* cold) migrate between hosts with different storage configurations. The reason is that the disks are pre-created blank on the destination, then qemu streams the bits over and converts them as they're written during the migration process. If you go down this route, though, you should plan to do it thoroughly so that at the end of the process you have a homogeneous storage configuration again. You should avoid doing any cold migrations at least in the intervening period.

[Josias Montag (monday)]

Thanks for your detailed explanation. I see the point. The confusing thing for me was that I saw the image_disk_format "raw" correctly stored in instance_system_metadata table. By this it seemed to me that nova should know that the disk is raw and not qcow2.

Nevertheless one does not expect that changing use_cow_images does break your existing, working instances. It might be a good idea to add a hint about this to the documentation / nova.conf.

[Matthew Booth (mbooth-9)]

image_disk_format refers to the original format of the image. It doesn't directly relate to the on-disk format used by the instance, as it will be converted if the compute host uses different storage.

[Sean Dague (sdague)]

Known limitation