subprocess.CalledProcessError: Command '['virsh', '-c', 'qemu:///system', 'secret-list']' returned non-zero exit

[Felipe Reyes]

According to the charm-deployment-guide[0] the recommended procedure to upgrade to a new OpenStack release is:

juju config nova-compute action-managed-upgrade=True
juju config nova-compute openstack-origin='cloud:xenial-queens'
juju run-action nova-compute/0 --wait pause
juju run-action nova-compute/0 --wait openstack-upgrade
juju run-action nova-compute/0 --wait resume

The problem with this is that the "pause" implementation for nova-compute stops the libvirt-bin daemon and then the openstack-upgrade action calls the config_changed() function[1] which will internally ends up calling "virsh" to talk to libvirt

2018-11-09 15:24:54 DEBUG openstack-upgrade error: failed to connect to the hypervisor
2018-11-09 15:24:54 DEBUG openstack-upgrade error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
2018-11-09 15:24:54 DEBUG openstack-upgrade Traceback (most recent call last):
2018-11-09 15:24:54 DEBUG openstack-upgrade File "/var/lib/juju/agents/unit-nova-compute-kvm-3/charm/actions/openstack-upgrade", line 47, in <module>
2018-11-09 15:24:54 DEBUG openstack-upgrade openstack_upgrade()
2018-11-09 15:24:54 DEBUG openstack-upgrade File "/var/lib/juju/agents/unit-nova-compute-kvm-3/charm/actions/openstack-upgrade", line 44, in openstack_upgrade
2018-11-09 15:24:54 DEBUG openstack-upgrade config_changed()
2018-11-09 15:24:54 DEBUG openstack-upgrade File "hooks/charmhelpers/contrib/openstack/utils.py", line 1462, in wrapped_f
2018-11-09 15:24:54 DEBUG openstack-upgrade return f(*args, **kwargs)
2018-11-09 15:24:54 DEBUG openstack-upgrade File "hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-11-09 15:24:54 DEBUG openstack-upgrade return f(*args, **kwargs)
2018-11-09 15:24:54 DEBUG openstack-upgrade File "hooks/nova_compute_hooks.py", line 220, in config_changed
2018-11-09 15:24:54 DEBUG openstack-upgrade ceph_changed(rid=rid, unit=unit)
2018-11-09 15:24:54 DEBUG openstack-upgrade File "hooks/charmhelpers/contrib/openstack/utils.py", line 1462, in wrapped_f
2018-11-09 15:24:54 DEBUG openstack-upgrade return f(*args, **kwargs)
2018-11-09 15:24:54 DEBUG openstack-upgrade File "hooks/nova_compute_hooks.py", line 376, in ceph_changed
2018-11-09 15:24:54 DEBUG openstack-upgrade secret_uuid=CEPH_SECRET_UUID, key=key)
2018-11-09 15:24:54 DEBUG openstack-upgrade File "hooks/nova_compute_utils.py", line 593, in create_libvirt_secret
2018-11-09 15:24:54 DEBUG openstack-upgrade if secret_uuid in check_output(cmd).decode('UTF-8'):
2018-11-09 15:24:54 DEBUG openstack-upgrade File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
2018-11-09 15:24:54 DEBUG openstack-upgrade **kwargs).stdout
2018-11-09 15:24:54 DEBUG openstack-upgrade File "/usr/lib/python3.5/subprocess.py", line 708, in run
2018-11-09 15:24:54 DEBUG openstack-upgrade output=stdout, stderr=stderr)
2018-11-09 15:24:54 DEBUG openstack-upgrade subprocess.CalledProcessError: Command '['virsh', '-c', 'qemu:///system', 'secret-list']' returned non-zero exit status 1

[0] https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-upgrade-openstack.html
[1] https://github.com/openstack/charm-nova-compute/blob/f60aa4a8fb3ece79a4be1b3280ba4f61645420ff/actions/openstack_upgrade.py#L44

[Felipe Reyes (freyes)]

The big problem I'm seeing now is that the pause_unit() uses the returned dict by register_config(), and it uses assess_status() to determine if all the expected services were actually stopped. So everything is tied together is not that simple as remove libvirt-bin daemon when calling pause_unit()

[James Page (james-page)]

I think we should exclude libvirt from the list of paused services; disabling nova-compute should be sufficient to ensure that openstack does not try to update anything remotely during the upgrade, so leaving libvirt running should be fine IMHO

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/624074

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/624074
Committed: https://git.openstack.org/cgit/openstack/charm-nova-compute/commit/?id=ceab1e91dc2e3948f6ba7c121c1801ad1641643c
Submitter: Zuul
Branch: master

commit ceab1e91dc2e3948f6ba7c121c1801ad1641643c
Author: Felipe Reyes <email address hidden>
Date: Mon Dec 10 09:52:43 2018 -0300

    Don't include libvirtd on pause/resume

    libvirt needs to be running even when the pause action runs, this allows
    commands like "virsh secret-list" run and having it running is safe
    enough from a nova-compute unit point of view, because it won't touch
    the already running instances.

    Change-Id: Idec9e2b7c6275c4c5485942574120289552f2955
    Closes-Bug: 1802917