nova can't access instance image file because the file is now chowned to the kvm group by default

[Takashi Kajinami]

[Impact]
This affects the nova package for kinetic and lunar. It is a side-effect of the changes made in https://bugs.launchpad.net/charm-nova-compute/+bug/1967956, specifically (1) and (3) described in https://bugs.launchpad.net/charm-nova-compute/+bug/1967956/comments/10. We tightened the mode of directories under /var/lib/nova from 755 to 750, and the mode of files under /var/lib/nova from 644 to 640. As a result, adding nova to the kvm group was required for nova to be able to access vm disks. We did that for nova-compute-kvm package but failed to do so for the nova-compute-qemu package.

== original bug description ==

It seems libvirt package in Ubuntu 22.04 uses the kvm group instead of the libvirt-qemu group when launching a qemu process.
Because of this change and the default behavior of libvirt which makes all image files chowned by the group/user to run qemu process, the instance files are owned by the kvm group, instead of the libvirt-qemu group.

However currently the nova user is still added to the libvirt-qemu group instead of the kvm group.

Because of this inconsistency nova can't access to instance image once the files are chowned to the kvm group.

nova 3:26.1.0-0ubuntu1~cloud0
libvirt 8.0.0-1ubuntu7.4

I've found the problem in puppet jobs. (example https://zuul.opendev.org/t/openstack/build/d6f2fb2e92ad4ece86bcd3d8793bf920 )
Example error in nova can be found here: https://2e4f6457af6d4bb29c73-cc818d493c2a52ef4d37701157d67702.ssl.cf2.rackcdn.com/877214/2/check/puppet-openstack-integration-7-scenario002-tempest-ubuntu-jammy/d6f2fb2/logs/nova/nova-compute.txt

I've tested adding the nova user to the kvm group and confirmed this fixes the error.
https://review.opendev.org/c/openstack/puppet-openstack-integration/+/877338

[Test Case]
At the most basic level we can install the nova-compute-qemu package on a machine, and install the nova-compute-kvm package on another machine, and compare the directory and file modes under the /var/lib/nova/ tree.

I'm sure Takashi will be able to give feedback on the fix as well.

[Regression Potential]
This is fixing a regression. We already add the nova user to the kvm group as part of the nova-compute-kvm postinst script, and this fix is doing the same for the nova-compute-qemu postinst script. I don't foresee any new regressions as a result of this.

[Corey Bryant (corey.bryant)]

Hi Takashi,

Thanks for reporting this. This code should be in place in the package:
https://git.launchpad.net/~ubuntu-openstack-dev/ubuntu/+source/nova/tree/debian/nova-compute-kvm.postinst?h=stable/zed
On your installed system you can find postinst scripts installed in /var/lib/dpkg/info/

Is it possible that you're using a different package version than you've reported?

Corey

[Takashi Kajinami (kajinamit)]

I think this is related to the fact that I'm using qemu instead of kvm.

https://2e4f6457af6d4bb29c73-cc818d493c2a52ef4d37701157d67702.ssl.cf2.rackcdn.com/877214/2/check/puppet-openstack-integration-7-scenario002-tempest-ubuntu-jammy/d6f2fb2/logs/etc/nova/nova.conf.txt
~~~
[libvirt]
virt_type=qemu
~~~

The node has nova-compute-qemu installed instead of nova-compute-kvm.

https://2e4f6457af6d4bb29c73-cc818d493c2a52ef4d37701157d67702.ssl.cf2.rackcdn.com/877214/2/check/puppet-openstack-integration-7-scenario002-tempest-ubuntu-jammy/d6f2fb2/logs/dpkg-l.txt
~~~
ii libvirt-clients 8.0.0-1ubuntu7.4 amd64 Programs for the libvirt library
ii libvirt-daemon 8.0.0-1ubuntu7.4 amd64 Virtualization daemon
ii libvirt-daemon-config-network 8.0.0-1ubuntu7.4 all Libvirt daemon configuration files (default network)
ii libvirt-daemon-config-nwfilter 8.0.0-1ubuntu7.4 all Libvirt daemon configuration files (default network filters)
ii libvirt-daemon-driver-qemu 8.0.0-1ubuntu7.4 amd64 Virtualization daemon QEMU connection driver
ii libvirt-daemon-system 8.0.0-1ubuntu7.4 amd64 Libvirt daemon configuration files
ii libvirt-daemon-system-systemd 8.0.0-1ubuntu7.4 all Libvirt daemon configuration files (systemd)
ii libvirt0:amd64 8.0.0-1ubuntu7.4 amd64 library for interfacing with different virtualization systems
...
ii nova-common 3:26.1.0-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 3:26.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node base
ii nova-compute-libvirt 3:26.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii nova-compute-qemu 3:26.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node (QEmu)
ii nova-conductor 3:26.1.0-0ubuntu1~cloud0 all OpenStack Compute - conductor service
ii nova-novncproxy 3:26.1.0-0ubuntu1~cloud0 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 3:26.1.0-0ubuntu1~cloud0 all OpenStack Compute - virtual machine scheduler
~~~

I suspect the change in libvirt (unintentionally or intentionally) changed the group used to run qemu process even if kvm is not used.

[Takashi Kajinami (kajinamit)]

IIUC this is the logic in libvirt build script to define the group to run the qemu process.

https://github.com/libvirt/libvirt/blob/8386242bd0f6c1cb242f9c711e2ef864bf114d0d/meson.build#L1607

[Corey Bryant (corey.bryant)]

Takashi, thanks for the additional details. This is a side-effect of these changes: https://bugs.launchpad.net/charm-nova-compute/+bug/1967956/comments/10

I'll update the qemu postinst script to add nova to the kvm group.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 3:26.1.0+git2023030309.59f7a524-0ubuntu2

---------------
nova (3:26.1.0+git2023030309.59f7a524-0ubuntu2) lunar; urgency=medium

  * d/nova-compute-qemu.postinst: Add nova user to kvm group (LP: #2011535).

 -- Corey Bryant <email address hidden> Wed, 15 Mar 2023 12:39:12 -0400

[Takashi Kajinami (kajinamit)]

Thanks. The fix sounds quite reasonable to me.

If you can backport the fix to UCA zed then that would be nice.
This is what we are using in the jobs in Puppet OpenStack project and
where I found the issue initially.

[Steve Langasek (vorlon)]

Hello Takashi, or anyone else affected,

Accepted nova into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/3:26.1.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

On kinetic-proposed and zed-proposed, I've verified that the nova user is added to the kvm group for the nova-compute-qemu package by inspecting /etc/group. Comparison of files under /var/lib/nova/ on separate machines with nova-compute-qemu package installed on one machine, and nova-compute-kvm package installed on the other machine, all file ownership matched.

[Corey Bryant (corey.bryant)]

Takashi, this is in zed-proposed now. Do you want to give it a try?

[Takashi Kajinami (kajinamit)]

I confirmed the issue is resolved by current zed-proposed packages. Thanks !

 https://review.opendev.org/c/openstack/puppet-openstack-integration/+/879991

[Corey Bryant (corey.bryant)]

Thanks for checking!

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package nova - 3:26.1.0-0ubuntu2~cloud0
---------------

 nova (3:26.1.0-0ubuntu2~cloud0) jammy-zed; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (3:26.1.0-0ubuntu2) kinetic; urgency=medium
 .
   * d/nova-compute-qemu.postinst: Add nova user to kvm group (LP: #2011535).

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 3:26.1.0-0ubuntu2

---------------
nova (3:26.1.0-0ubuntu2) kinetic; urgency=medium

  * d/nova-compute-qemu.postinst: Add nova user to kvm group (LP: #2011535).

 -- Corey Bryant <email address hidden> Wed, 15 Mar 2023 12:42:12 -0400