instances-path config does not work properly

[Rodrigo Barbieri]

I tried changing the config "instances-path" to set path /nova/my_instances instead of the default /var/lib/nova/instances. Upon trying to create an instance, it fails to build and the stack trace below is found on nova-compute logs.

I realized that I had to create the path "/nova/my_instances" myself, as the charm doesn't create and it produces a different error. Also, I tried to address the permission denied error using "Chown -R /nova/my_instances", but still had no success. The charm actually does run chown so this shouldn't be needed.

After the error, I tried comparing it with the permissions of my default /var/lib/nova/instances, and the permission bits seemed to be the same.

Although there is a slightly similar bug 1656254, in my testing I was not using apparmor nor encryption.

2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [req-202c3fb4-a4e7-4bb3-aca3-b8c3f4c3f81a e78c033f8d974a27aeee94536d6afb1f 867f0965fdd4432f8e11517121b2ea81 - 54bdddc7db4547e1bf5d6f8651cb86fc 54bdddc7db4547e1bf5d6f8651cb86fc] [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] Instance failed to spawn: libvirtError: internal error: process exited while connecting to monitor: 2020-03-08T18:03:21.198569Z qemu-system-x86_64: -drive file=/nova/my_instances/c0b5e6f5-5936-4de5-b10d-48ac409ff434/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,discard=unmap: Could not open backing file: Could not open '/nova/my_instances/_base/8281f19ac8157de59a7df0860a99858960e2f8f0': Permission denied
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] Traceback (most recent call last):
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2274, in _build_resources
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] yield resources
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2054, in _build_and_run_instance
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] block_device_info=block_device_info)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3147, in spawn
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] destroy_disks_on_failure=True)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5651, in _create_domain_and_network
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] destroy_disks_on_failure)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] self.force_reraise()
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] six.reraise(self.type_, self.value, self.tb)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5620, in _create_domain_and_network
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] post_xml_callback=post_xml_callback)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5555, in _create_domain
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] guest.launch(pause=pause)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 144, in launch
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] self._encoded_xml, errors='ignore')
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] self.force_reraise()
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] six.reraise(self.type_, self.value, self.tb)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 139, in launch
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] return self._domain.createWithFlags(flags)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] rv = execute(f, *args, **kwargs)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] six.reraise(c, e, tb)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] rv = meth(*args, **kwargs)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1092, in createWithFlags
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] libvirtError: internal error: process exited while connecting to monitor: 2020-03-08T18:03:21.198569Z qemu-system-x86_64: -drive file=/nova/my_instances/c0b5e6f5-5936-4de5-b10d-48ac409ff434/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,discard=unmap: Could not open backing file: Could not open '/nova/my_instances/_base/8281f19ac8157de59a7df0860a99858960e2f8f0': Permission denied
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434]
2020-03-08 18:03:21.934 18406 INFO nova.compute.manager [req-202c3fb4-a4e7-4bb3-aca3-b8c3f4c3f81a e78c033f8d974a27aeee94536d6afb1f 867f0965fdd4432f8e11517121b2ea81 - 54bdddc7db4547e1bf5d6f8651cb86fc 54bdddc7db4547e1bf5d6f8651cb86fc] [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] Terminating instance

[Andrew McLeod (admcleod)]

Thanks for your bug report - in order to replicate this we need to know which distro and if you used specific charm revisions? An example bundle may also be helpful.

[Rodrigo Barbieri (rodrigo-barbieri2010)]

Hello Andrew,

I attached the output of my juju export-bundle. Version used was bionic-queens.

[Rodrigo Barbieri (rodrigo-barbieri2010)]

I tested editing the libvirt apparmor profile /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper and it did indeed address the issue. I started coding a way to have the charm update that file, however I haven't figured out yet a way for the charm to reliably maintain its values.

The file's default path is /var/lib/nova/instances. Suppose the file is edited and the value "/var/lib/nova/instances" is not longer there, but the new path. The next instances-path config editing will need to update that accordingly.

Even with Config.previous, we run the risk of the file and the config going offsync if the charm hook fails.

The file is also a bit fragile in the sense that package updates may overwrite and cause nova to stop working and a config-changed hook or some periodic validation would need to kick in to address the problem.

Another point of fragility is that since there is no tag to specific lines, if a later update changes those specific lines of the file, it will break the charm and require an update.

[Rodrigo Barbieri (rodrigo-barbieri2010)]

we had a WIP patch proposed for this a long time ago (I had forgotten to include Closes-bug in the commit msg because it was WIP and it ended up not linked here).

https://review.opendev.org/c/openstack/charm-nova-compute/+/718219

At the time we had lots of internal discussions and we ended up not moving forward with it. We currently do not have a meaningful demand for this anymore so the patch has been abandoned.