File permissions in /var/lib/nova/.ssh broken in upgrade

[Jake Hill]

I am doing series upgrade on charmed openstack following the normal procedure. The upgrade is cloud:bionic-ussuri to focal distro.

On compute units I notice that running apt full-upgrade prior to doing do-release-upgrade results in incorrect file permissions on /var/lib/nova/.ssh/. This in turn breaks migrations at least.

I did not catch if it was nova-common or nova-compute that did this during the upgrade, but something wrote 644 on everything.

(osc) routergod@juju:~$ juju ssh 40 -- sudo ls -l /var/lib/nova/.ssh
total 44
-rw-r----- 1 nova nova 10615 Nov 10 17:34 authorized_keys
-rw------- 1 nova nova 1675 Apr 9 2020 id_rsa
-rw-r----- 1 nova nova 393 Apr 9 2020 id_rsa.pub
-rw-r----- 1 nova nova 21216 Nov 10 17:34 known_hosts
(osc) routergod@juju:~$ juju upgrade-series 40 prepare focal
WARNING: This command will mark machine "40" as being upgraded to series "focal".
This operation cannot be reverted or canceled once started.
Units running on the machine will also be upgraded. These units include:
  nova-compute/11
  ntp/151
  neutron-openvswitch/67
  nova-compute-syslog/0
[...]
(osc) routergod@juju:~$ juju ssh 40 -- sudo apt update
[...]
(osc) routergod@juju:~$ juju ssh 40 -- sudo apt full-upgrade
[...]
(osc) routergod@juju:~$ juju ssh 40 -- sudo ls -l /var/lib/nova/.ssh
total 44
-rw-r--r-- 1 nova nova 10615 Nov 10 17:34 authorized_keys
-rw-r--r-- 1 nova nova 1675 Apr 9 2020 id_rsa
-rw-r--r-- 1 nova nova 393 Apr 9 2020 id_rsa.pub
-rw-r--r-- 1 nova nova 21216 Nov 10 17:34 known_hosts

[Jake Hill (routergod)]

FWIW and perhaps expectedly, the subsequent do-release-upgrade step also mangles the permissions.

[Alex Dodson (alexdodson)]

I am seeing this issue on a straight focal install with no upgrade from bionic. id_rsa has perms set 0644 and i have to ssh onto the hypervisor and reset them to 0600

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alex Dodson (alexdodson)]

It occurred for me on an apt upgrade to the nova patch level of nova-compute-kvm/focal-updates,now 2:21.2.1-0ubuntu1 all [installed]

[Felipe Reyes (freyes)]

hello everyone, thanks for reporting this bug, I'm going to mark this bug as duplicate of https://bugs.launchpad.net/charm-nova-compute/+bug/1904580 since it tracked the analysis and workarounds.