subprocess.CalledProcessError: Command '['virsh', '-c', 'qemu:///system', 'secret-list']' returned non-zero exit

Bug #1802917 reported by Felipe Reyes
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Nova Compute Charm
Fix Released
Medium
Felipe Reyes

Bug Description

According to the charm-deployment-guide[0] the recommended procedure to upgrade to a new OpenStack release is:

juju config nova-compute action-managed-upgrade=True
juju config nova-compute openstack-origin='cloud:xenial-queens'
juju run-action nova-compute/0 --wait pause
juju run-action nova-compute/0 --wait openstack-upgrade
juju run-action nova-compute/0 --wait resume

The problem with this is that the "pause" implementation for nova-compute stops the libvirt-bin daemon and then the openstack-upgrade action calls the config_changed() function[1] which will internally ends up calling "virsh" to talk to libvirt

2018-11-09 15:24:54 DEBUG openstack-upgrade error: failed to connect to the hypervisor
2018-11-09 15:24:54 DEBUG openstack-upgrade error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
2018-11-09 15:24:54 DEBUG openstack-upgrade Traceback (most recent call last):
2018-11-09 15:24:54 DEBUG openstack-upgrade File "/var/lib/juju/agents/unit-nova-compute-kvm-3/charm/actions/openstack-upgrade", line 47, in <module>
2018-11-09 15:24:54 DEBUG openstack-upgrade openstack_upgrade()
2018-11-09 15:24:54 DEBUG openstack-upgrade File "/var/lib/juju/agents/unit-nova-compute-kvm-3/charm/actions/openstack-upgrade", line 44, in openstack_upgrade
2018-11-09 15:24:54 DEBUG openstack-upgrade config_changed()
2018-11-09 15:24:54 DEBUG openstack-upgrade File "hooks/charmhelpers/contrib/openstack/utils.py", line 1462, in wrapped_f
2018-11-09 15:24:54 DEBUG openstack-upgrade return f(*args, **kwargs)
2018-11-09 15:24:54 DEBUG openstack-upgrade File "hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-11-09 15:24:54 DEBUG openstack-upgrade return f(*args, **kwargs)
2018-11-09 15:24:54 DEBUG openstack-upgrade File "hooks/nova_compute_hooks.py", line 220, in config_changed
2018-11-09 15:24:54 DEBUG openstack-upgrade ceph_changed(rid=rid, unit=unit)
2018-11-09 15:24:54 DEBUG openstack-upgrade File "hooks/charmhelpers/contrib/openstack/utils.py", line 1462, in wrapped_f
2018-11-09 15:24:54 DEBUG openstack-upgrade return f(*args, **kwargs)
2018-11-09 15:24:54 DEBUG openstack-upgrade File "hooks/nova_compute_hooks.py", line 376, in ceph_changed
2018-11-09 15:24:54 DEBUG openstack-upgrade secret_uuid=CEPH_SECRET_UUID, key=key)
2018-11-09 15:24:54 DEBUG openstack-upgrade File "hooks/nova_compute_utils.py", line 593, in create_libvirt_secret
2018-11-09 15:24:54 DEBUG openstack-upgrade if secret_uuid in check_output(cmd).decode('UTF-8'):
2018-11-09 15:24:54 DEBUG openstack-upgrade File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
2018-11-09 15:24:54 DEBUG openstack-upgrade **kwargs).stdout
2018-11-09 15:24:54 DEBUG openstack-upgrade File "/usr/lib/python3.5/subprocess.py", line 708, in run
2018-11-09 15:24:54 DEBUG openstack-upgrade output=stdout, stderr=stderr)
2018-11-09 15:24:54 DEBUG openstack-upgrade subprocess.CalledProcessError: Command '['virsh', '-c', 'qemu:///system', 'secret-list']' returned non-zero exit status 1

[0] https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-upgrade-openstack.html
[1] https://github.com/openstack/charm-nova-compute/blob/f60aa4a8fb3ece79a4be1b3280ba4f61645420ff/actions/openstack_upgrade.py#L44

Tags: sts
Felipe Reyes (freyes)
tags: added: sts
Revision history for this message
Felipe Reyes (freyes) wrote :

The big problem I'm seeing now is that the pause_unit() uses the returned dict by register_config(), and it uses assess_status() to determine if all the expected services were actually stopped. So everything is tied together is not that simple as remove libvirt-bin daemon when calling pause_unit()

Revision history for this message
James Page (james-page) wrote :

I think we should exclude libvirt from the list of paused services; disabling nova-compute should be sufficient to ensure that openstack does not try to update anything remotely during the upgrade, so leaving libvirt running should be fine IMHO

Changed in charm-nova-compute:
status: New → Triaged
importance: Undecided → Medium
Felipe Reyes (freyes)
Changed in charm-nova-compute:
assignee: nobody → Felipe Reyes (freyes)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-compute (master)

Fix proposed to branch: master
Review: https://review.openstack.org/624074

Changed in charm-nova-compute:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-compute (master)

Reviewed: https://review.openstack.org/624074
Committed: https://git.openstack.org/cgit/openstack/charm-nova-compute/commit/?id=ceab1e91dc2e3948f6ba7c121c1801ad1641643c
Submitter: Zuul
Branch: master

commit ceab1e91dc2e3948f6ba7c121c1801ad1641643c
Author: Felipe Reyes <email address hidden>
Date: Mon Dec 10 09:52:43 2018 -0300

    Don't include libvirtd on pause/resume

    libvirt needs to be running even when the pause action runs, this allows
    commands like "virsh secret-list" run and having it running is safe
    enough from a nova-compute unit point of view, because it won't touch
    the already running instances.

    Change-Id: Idec9e2b7c6275c4c5485942574120289552f2955
    Closes-Bug: 1802917

Changed in charm-nova-compute:
status: In Progress → Fix Committed
James Page (james-page)
Changed in charm-nova-compute:
milestone: none → 19.04
David Ames (thedac)
Changed in charm-nova-compute:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.