ussuri: unable to stop and start an instance post upgrade

[James Page]

== Impact ==

instances cannot be stopped and then started

== Test Case ==

Deploy OpenStack (using Juju Charms)
Create networking and boot an instance
stop the instance
start the instance
(fails with permissions errors)

== Regression Potential ==

Low - the proposed change opens the permission under /var/lib/nova to allow for the odd ownership changes that happen during instance lifecycle.

== Original Bug Report ==

bionic or focal with OpenStack Ussuri.

The latest version of the nova package sets permissions on package update for /var/lib/nova:

  find /var/lib/nova -exec chown nova:nova "{}" +
  find /var/lib/nova -type f -exec chmod 0640 "{}" + -o -type d -exec chmod 0750 "{}" +

However, when managing vm's via libvirt, various ownership changes happen to the underlying disks that mean that nova can no longer access the disk files with 0640 permissions.

The disks (and base image) for a vm are created as nova:nova; libvirt then shifts the ownership to libvirt-qemu:kvm as the vm starts. When the vm is stopped the ownership reverts to root:root.

0640 permissions are maintained - however nova is not part of the root or kvm groups so cannot access the file - which means the instance cannot be restarted.

The following permissions are required for correct operation:

  find /var/lib/nova -type f -exec chmod 0644 "{}" + -o -type d -exec chmod 0755 "{}" +

[James Page (james-page)]

Linked bug 691590

[Alex Kavanagh (ajkavanagh)]

Additional details from a compute unit:

on nova/compute/1 -- before stopping the unit, but after upgrading from train->ussuri on bionic. Note that the disk, console.log are owned nova.nova.

root@juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7# ll
total 184692
drwxr-x--- 2 nova nova 4096 Jun 25 17:46 ./
drwxr-x--- 5 nova nova 4096 Jun 25 18:21 ../
-rw-r----- 1 nova nova 31316 Jun 25 17:47 console.log
-rw-r----- 1 nova nova 189136896 Jun 26 10:17 disk
-rw-r----- 1 nova nova 79 Jun 25 17:46 disk.info
root@juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90

After stopping the instance, the disk is now owned root.root. As there is no world-readable perm, nova now can't restart the instance.

root@juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7# ll
total 184948
drwxr-x--- 2 nova nova 4096 Jun 25 17:46 ./
drwxr-x--- 5 nova nova 4096 Jun 25 18:21 ../
-rw-r----- 1 nova nova 31316 Jun 25 17:47 console.log
-rw-r----- 1 root root 189399040 Jun 26 10:38 disk
-rw-r----- 1 nova nova 79 Jun 25 17:46 disk.info

After fixing the permissions back to nova.nova and restarting the unit, the console.log is now owned by root, and the disk by libvert-qumu.kvm.

root@juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7# ll
total 184924
drwxr-x--- 2 nova nova 4096 Jun 26 10:39 ./
drwxr-x--- 5 nova nova 4096 Jun 26 10:38 ../
-rw------- 1 root root 8039 Jun 26 10:39 console.log
-rw-r----- 1 libvirt-qemu kvm 189399040 Jun 26 10:39 disk
-rw-r----- 1 nova nova 79 Jun 25 17:46 disk.info

When this unit is stopped, it again goes back to root.root.

root@juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7# ll
total 326324
drwxr-x--- 2 nova nova 4096 Jun 26 10:39 ./
drwxr-x--- 5 nova nova 4096 Jun 26 10:38 ../
-rw------- 1 root root 29040 Jun 26 10:41 console.log
-rw-r----- 1 root root 334168064 Jun 26 10:41 disk
-rw-r----- 1 nova nova 79 Jun 25 17:46 disk.info
root@juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7#

The only real solution, for the moment, is to revert the permissions (as indicated in the summary) back to world readable for the disk and ../_disk/* files.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 2:21.0.0-0ubuntu3

---------------
nova (2:21.0.0-0ubuntu3) groovy; urgency=medium

  * d/rules: Blacklist test_tcp_rst_no_compute_rpcapi.

 -- James Page <email address hidden> Tue, 30 Jun 2020 09:06:12 +0100

[James Page (james-page)]

nova uploaded to focal-proposed for SRU team review.

[Łukasz Zemczak (sil2100)]

Hello James, or anyone else affected,

Accepted nova into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:21.0.0-0ubuntu0.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[James Page (james-page)]

Verification completed - able to stop and then start an instance using the package from proposed.

[James Page (james-page)]

verification completed in bionic-ussuri/proposed (although there are not comments or tags to manage here). Able to stop and then start an instance using the proposed package for nova.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package nova - 2:22.0.0~b1~git2020070713.bc784a1c1f-0ubuntu1~cloud0
---------------

 nova (2:22.0.0~b1~git2020070713.bc784a1c1f-0ubuntu1~cloud0) focal-victoria; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 nova (2:22.0.0~b1~git2020070713.bc784a1c1f-0ubuntu1) groovy; urgency=medium
 .
   * New upstream snapshot for OpenStack Victoria.
   * d/control: Align (Build-)Depends with upstream.
   * d/p/add-mysql8-compatibility.patch: Removed. Change landed upstream.
   * d/p/arm-console-patch.patch: Refreshed.
   * d/p/drop-sphinxcontrib-rsvgconverter.patch: Refreshed
 .
 nova (2:21.0.0-0ubuntu3) groovy; urgency=medium
 .
   * d/rules: Blacklist test_tcp_rst_no_compute_rpcapi.
 .
 nova (2:21.0.0-0ubuntu2) groovy; urgency=medium
 .
   * Correct file and directory permissions to ensure that instances
     can be stopped and started correctly (LP: #1885269):
     - d/nova-common.postinst: Reset file permissions under /var/lib/nova
       to 0644 and directory permissions to 0755 to ensure that instances
       can be stopped and started correctly with the workflow of ownership
       changes that occur between nova and libvirt/qemu.
     - d/nova-common.postinst: Don't force ownership to nova:nova.
     - d/nova-common.postinst: Reset ownership and permissions on
       console.log files to root:root and 0600.
 .
 nova (2:21.0.0-0ubuntu1) groovy; urgency=medium
 .
   * d/watch: Scope to 21.x series and get tarballs from opendev.org.
   * d/control: Align (Build-)Depends with upstream.
   * New upstream release for OpenStack Ussuri (LP: #1877642).

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 2:21.0.0-0ubuntu0.20.04.2

---------------
nova (2:21.0.0-0ubuntu0.20.04.2) focal; urgency=medium

  * Correct file and directory permissions to ensure that instances
    can be stopped and started correctly (LP: #1885269):
    - d/nova-common.postinst: Reset file permissions under /var/lib/nova
      to 0644 and directory permissions to 0755 to ensure that instances
      can be stopped and started correctly with the workflow of ownership
      changes that occur between nova and libvirt/qemu.
    - d/nova-common.postinst: Don't force ownership to nova:nova.
    - d/nova-common.postinst: Reset ownership and permissions on
      console.log files to root:root and 0600.

 -- James Page <email address hidden> Mon, 29 Jun 2020 13:54:40 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for nova has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.