TLDR: nova is allowing back VMDK with wrong types and QCOW with backing files.
Long:
The following steps were used to reproduce on a Bobcat (2023.2) OpenStack + nova patches for bug #2059809 (not yet merge when writing this report)
Create a vmdk file:
$ qemu-img create -f vmdk disk-vmdk.vmdk 1M -o subformat=monolithicFlat
$ sed -i -r 's|disk-vmdk-flat.vmdk|/etc/hosts|' disk-vmdk.vmdk
Create a faulty qcow image:
$ qemu-img create -f qcow2 -F raw -b /etc/hosts disk-bf.qcow2 1M
Upload both images as raw (the default)
$ for i in disk-bf.qcow2 disk-vmdk.vmdk ; do openstack image create --file $i $i ; done
Boot an instance from those images:
$ openstack server create --flavor small --image disk-vmdk.vmdk --net public disk-vmdk.vmdk
$ openstack server create --flavor small --image disk-bf.qcow2 --net public disk-bf.qcow2
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2615, in _build_and_run_instance
self.driver.spawn(context, instance, image_meta,
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4388, in spawn
created_instance_dir, created_disks = self._create_image( ^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4790, in _create_image
created_disks = self._create_and_inject_local_root( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4921, in _create_and_inject_local_root
self._try_fetch_image_cache(backend, fetch_func, context,
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 10940, in _try_fetch_image_cache
image.cache(fetch_func=fetch_func,
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py", line 288, in cache
self.create_image(fetch_func_sync, base, size,
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py", line 615, in create_image
copy_raw_image(base, self.path, size)
File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 414, in inner
return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py", line 590, in copy_raw_image
self.resize_image(size)
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py", line 621, in resize_image
disk.extend(image, size)
File "/usr/lib/python3/dist-packages/nova/virt/disk/api.py", line 128, in extend
processutils.execute('qemu-img', 'resize', image.path, size)
File "/usr/lib/python3/dist-packages/oslo_concurrency/processutils.py", line 438, in execute
raise ProcessExecutionError(exit_code=_returncode,
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: qemu-img resize /var/lib/nova/instances/07b5907b-5efc-4fdf-8b15-4b47a820c2f8/disk 2147483648
Exit code: 1
Stdout: ''
Stderr: "qemu-img: Could not open '/var/lib/nova/instances/07b5907b-5efc-4fdf-8b15-4b47a820c2f8/disk': Could not open '/etc/hosts': Permission denied\n"
Qemu tried to read /etc/hosts. My system permissions prevented it, but nova did nothing about it: wrong.
QEMU Backing File
-----------------
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2615, in _build_and_run_instance
self.driver.spawn(context, instance, image_meta,
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4415, in spawn
self._create_guest_with_network(
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7785, in _create_guest_with_network
with excutils.save_and_reraise_exception():
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
self.force_reraise()
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
raise self.value
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7763, in _create_guest_with_network
guest = self._create_guest( ^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7702, in _create_guest
guest.launch(pause=pause)
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 167, in launch
with excutils.save_and_reraise_exception():
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
self.force_reraise()
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
raise self.value
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 165, in launch
return self._domain.createWithFlags(flags) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
rv = execute(f, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
six.reraise(c, e, tb)
File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
raise value
File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
rv = meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/libvirt.py", line 1409, in createWithFlags
raise libvirtError('virDomainCreateWithFlags() failed')
libvirt.libvirtError: internal error: cannot update AppArmor profile 'libvirt-6bd32822-2454-402a-9617-6ec66e0090f4'
Here it's apparmor that prevented the boot, but nova should have catched it: wrong
Expected results
----------------
Nova should raise an exception like it does previously.
E.G. for VMDK: nova.exception.ImageUnacceptable: Image xyz is unacceptable: Invalid VMDK create-type specified
When fixing bug #2059809, a regression of the previous bug #1996188 has been introduced.
TLDR: nova is allowing back VMDK with wrong types and QCOW with backing files.
Long:
The following steps were used to reproduce on a Bobcat (2023.2) OpenStack + nova patches for bug #2059809 (not yet merge when writing this report)
Create a vmdk file: monolithicFlat vmdk-flat. vmdk|/etc/ hosts|' disk-vmdk.vmdk
$ qemu-img create -f vmdk disk-vmdk.vmdk 1M -o subformat=
$ sed -i -r 's|disk-
Create a faulty qcow image:
$ qemu-img create -f qcow2 -F raw -b /etc/hosts disk-bf.qcow2 1M
Upload both images as raw (the default)
$ for i in disk-bf.qcow2 disk-vmdk.vmdk ; do openstack image create --file $i $i ; done
Boot an instance from those images:
$ openstack server create --flavor small --image disk-vmdk.vmdk --net public disk-vmdk.vmdk
$ openstack server create --flavor small --image disk-bf.qcow2 --net public disk-bf.qcow2
Results
=======
VMDK monolithicFlat
-------------------
Traceback (most recent call last): python3/ dist-packages/ nova/compute/ manager. py", line 2615, in _build_ and_run_ instance driver. spawn(context, instance, image_meta, python3/ dist-packages/ nova/virt/ libvirt/ driver. py", line 4388, in spawn instance_ dir, created_disks = self._create_image(
^^^^^^^ ^^^^^^^ ^^^^^ python3/ dist-packages/ nova/virt/ libvirt/ driver. py", line 4790, in _create_image and_inject_ local_root(
^ ^^^^^^^ ^^^^^^^ ^^^^^^^ ^^^^^^^ ^^^^^^ python3/ dist-packages/ nova/virt/ libvirt/ driver. py", line 4921, in _create_ and_inject_ local_root _try_fetch_ image_cache( backend, fetch_func, context, python3/ dist-packages/ nova/virt/ libvirt/ driver. py", line 10940, in _try_fetch_ image_cache cache(fetch_ func=fetch_ func, python3/ dist-packages/ nova/virt/ libvirt/ imagebackend. py", line 288, in cache create_ image(fetch_ func_sync, base, size, python3/ dist-packages/ nova/virt/ libvirt/ imagebackend. py", line 615, in create_image raw_image( base, self.path, size) python3/ dist-packages/ oslo_concurrenc y/lockutils. py", line 414, in inner
^^^ ^^^^^^^ ^^^^^^^ ^ python3/ dist-packages/ nova/virt/ libvirt/ imagebackend. py", line 590, in copy_raw_image resize_ image(size) python3/ dist-packages/ nova/virt/ libvirt/ imagebackend. py", line 621, in resize_image extend( image, size) python3/ dist-packages/ nova/virt/ disk/api. py", line 128, in extend s.execute( 'qemu-img' , 'resize', image.path, size) python3/ dist-packages/ oslo_concurrenc y/processutils. py", line 438, in execute nError( exit_code= _returncode, y.processutils. ProcessExecutio nError: Unexpected error while running command. nova/instances/ 07b5907b- 5efc-4fdf- 8b15-4b47a820c2 f8/disk 2147483648 nova/instances/ 07b5907b- 5efc-4fdf- 8b15-4b47a820c2 f8/disk' : Could not open '/etc/hosts': Permission denied\n"
File "/usr/lib/
self.
File "/usr/lib/
created_
File "/usr/lib/
created_disks = self._create_
File "/usr/lib/
self.
File "/usr/lib/
image.
File "/usr/lib/
self.
File "/usr/lib/
copy_
File "/usr/lib/
return f(*args, **kwargs)
File "/usr/lib/
self.
File "/usr/lib/
disk.
File "/usr/lib/
processutil
File "/usr/lib/
raise ProcessExecutio
oslo_concurrenc
Command: qemu-img resize /var/lib/
Exit code: 1
Stdout: ''
Stderr: "qemu-img: Could not open '/var/lib/
Qemu tried to read /etc/hosts. My system permissions prevented it, but nova did nothing about it: wrong.
QEMU Backing File python3/ dist-packages/ nova/compute/ manager. py", line 2615, in _build_ and_run_ instance driver. spawn(context, instance, image_meta, python3/ dist-packages/ nova/virt/ libvirt/ driver. py", line 4415, in spawn _create_ guest_with_ network( python3/ dist-packages/ nova/virt/ libvirt/ driver. py", line 7785, in _create_ guest_with_ network save_and_ reraise_ exception( ): python3/ dist-packages/ oslo_utils/ excutils. py", line 227, in __exit__ force_reraise( ) python3/ dist-packages/ oslo_utils/ excutils. py", line 200, in force_reraise python3/ dist-packages/ nova/virt/ libvirt/ driver. py", line 7763, in _create_ guest_with_ network
^^ ^^^^^^^ ^^^^^^^ ^^^ python3/ dist-packages/ nova/virt/ libvirt/ driver. py", line 7702, in _create_guest launch( pause=pause) python3/ dist-packages/ nova/virt/ libvirt/ guest.py" , line 167, in launch save_and_ reraise_ exception( ): python3/ dist-packages/ oslo_utils/ excutils. py", line 227, in __exit__ force_reraise( ) python3/ dist-packages/ oslo_utils/ excutils. py", line 200, in force_reraise python3/ dist-packages/ nova/virt/ libvirt/ guest.py" , line 165, in launch createWithFlags (flags)
^^^ ^^^^^^^ ^^^^^^^ ^^^^^^^ ^^^^^^^ ^^^^ python3/ dist-packages/ eventlet/ tpool.py" , line 193, in doit self._autowrap, f, *args, **kwargs)
^ ^^^^^^^ ^^^^^^^ ^^^^^^^ ^^^^^^^ ^^^^^^^ ^^^^^^^ ^^^ python3/ dist-packages/ eventlet/ tpool.py" , line 151, in proxy_call
^^^^^ ^^^^^^^ ^^^^^^^ ^^^^^^^ ^ python3/ dist-packages/ eventlet/ tpool.py" , line 132, in execute python3/ dist-packages/ six.py" , line 719, in reraise python3/ dist-packages/ eventlet/ tpool.py" , line 86, in tworker
^^^^^ ^^^^^^^ ^^^^^^^ ^^ python3/ dist-packages/ libvirt. py", line 1409, in createWithFlags 'virDomainCreat eWithFlags( ) failed') libvirtError: internal error: cannot update AppArmor profile 'libvirt- 6bd32822- 2454-402a- 9617-6ec66e0090 f4'
-----------------
Traceback (most recent call last):
File "/usr/lib/
self.
File "/usr/lib/
self.
File "/usr/lib/
with excutils.
File "/usr/lib/
self.
File "/usr/lib/
raise self.value
File "/usr/lib/
guest = self._create_guest(
File "/usr/lib/
guest.
File "/usr/lib/
with excutils.
File "/usr/lib/
self.
File "/usr/lib/
raise self.value
File "/usr/lib/
return self._domain.
File "/usr/lib/
result = proxy_call(
File "/usr/lib/
rv = execute(f, *args, **kwargs)
File "/usr/lib/
six.reraise(c, e, tb)
File "/usr/lib/
raise value
File "/usr/lib/
rv = meth(*args, **kwargs)
File "/usr/lib/
raise libvirtError(
libvirt.
In libvirtd journalctl:
Jul 02 20:22:44 compute-1 libvirtd[959438]: internal error: Child process (LIBVIRT_ LOG_OUTPUTS= 3:stderr /usr/lib/ libvirt/ virt-aa- helper -r -u libvirt- 6bd32822- 2454-402a- 9617-6ec66e0090 f4 -F /dev/net/tun) unexpected exit status 1: virt-aa-helper: error: /etc/hosts
Here it's apparmor that prevented the boot, but nova should have catched it: wrong
Expected results
----------------
Nova should raise an exception like it does previously. ImageUnacceptab le: Image xyz is unacceptable: Invalid VMDK create-type specified
E.G. for VMDK: nova.exception.