hook failed: "secrets-storage-relation-changed" caused by vaultlocker.py trying to import hvac, but hvac is not available (ModuleNotFoundError: No module named 'hvac')

[Przemyslaw Hausman]

DEPLOYMENT

bionic-stein, DVR, barbican + octavia, external Ceph cluster (ceph-proxy charm deployed), Vault.

PROBLEM DESCRIPTION

As soon as Vault is unlocked, all nova-compute-kvm units go into 'error' state with status message 'hook failed: "secrets-storage-relation-changed"'.

Inspecting juju debug-log of nova-compute-kvm units I find the following error:

unit-nova-compute-kvm-1: 23:19:52 INFO unit.nova-compute-kvm/1.juju-log secrets-storage:277: Making dir /var/lib/charm/nova-compute-kvm root:root 555
unit-nova-compute-kvm-1: 23:19:52 INFO unit.nova-compute-kvm/1.juju-log secrets-storage:277: Making dir /etc/ceph root:root 555
unit-nova-compute-kvm-1: 23:19:53 INFO unit.nova-compute-kvm/1.juju-log secrets-storage:277: Registered config file: /etc/libvirt/qemu.conf
unit-nova-compute-kvm-1: 23:19:53 INFO unit.nova-compute-kvm/1.juju-log secrets-storage:277: Registered config file: /etc/default/qemu-kvm
unit-nova-compute-kvm-1: 23:19:53 INFO unit.nova-compute-kvm/1.juju-log secrets-storage:277: Registered config file: /etc/libvirt/libvirtd.conf
unit-nova-compute-kvm-1: 23:19:53 INFO unit.nova-compute-kvm/1.juju-log secrets-storage:277: Registered config file: /etc/default/libvirt-bin
unit-nova-compute-kvm-1: 23:19:53 INFO unit.nova-compute-kvm/1.juju-log secrets-storage:277: Registered config file: /etc/init/libvirt-bin.override
unit-nova-compute-kvm-1: 23:19:53 INFO unit.nova-compute-kvm/1.juju-log secrets-storage:277: Registered config file: /etc/nova/nova.conf
unit-nova-compute-kvm-1: 23:19:53 INFO unit.nova-compute-kvm/1.juju-log secrets-storage:277: Registered config file: /etc/nova/vendor_data.json
unit-nova-compute-kvm-1: 23:19:53 INFO unit.nova-compute-kvm/1.juju-log secrets-storage:277: Registered config file: /etc/apparmor.d/usr.bin.nova-compute
unit-nova-compute-kvm-1: 23:19:53 INFO unit.nova-compute-kvm/1.juju-log secrets-storage:277: Registered config file: /etc/ceph/secret.xml
unit-nova-compute-kvm-1: 23:19:53 INFO unit.nova-compute-kvm/1.juju-log secrets-storage:277: Registered config file: /var/lib/charm/nova-compute-kvm/ceph.conf
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed Traceback (most recent call last):
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-1/charm/hooks/secrets-storage-relation-changed", line 763, in <module>
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed main()
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-1/charm/hooks/secrets-storage-relation-changed", line 756, in main
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed hooks.execute(sys.argv)
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-1/charm/hooks/charmhelpers/core/hookenv.py", line 914, in execute
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed self._hooks[hook_name]()
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-1/charm/hooks/secrets-storage-relation-changed", line 702, in secrets_storage_changed
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed configure_local_ephemeral_storage()
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-1/charm/hooks/nova_compute_utils.py", line 973, in configure_local_ephemeral_storage
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed context = vault_kv()
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-1/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 60, in __call__
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed token=token
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-1/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 121, in retrieve_secret_id
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed import hvac
unit-nova-compute-kvm-1: 23:19:53 DEBUG unit.nova-compute-kvm/1.secrets-storage-relation-changed ModuleNotFoundError: No module named 'hvac'
unit-nova-compute-kvm-1: 23:19:53 ERROR juju.worker.uniter.operation hook "secrets-storage-relation-changed" failed: exit status 1

WORKAROUND

1. Manually install python3-hvac on nova-compute-kvm nodes (apt install python3-hvac)
2. juju resolved nova-compute-kvm/N
3. nova-compute-kvm units go into 'active' state with status message 'Unit is ready'

BUNDLE

https://pastebin.canonical.com/p/7cXhnHXZhM/

CHARMS REVISIONS

https://pastebin.canonical.com/p/mhprgzbkw7/

[Przemyslaw Hausman (phausman)]

During discussion with Alex, we discovered that my bundle contains unnecessary relation:

  "vault:secrets" - "nova-compute-kvm:secrets-storage"

The relation is not needed, because encryption of ephemeral storage is disabled (encrypt: false).

After removing the relation, the error goes away.

[Alex Kavanagh (ajkavanagh)]

Thanks for the update Przemyslaw. There is a bug still, though, and it's in the robustness of the charm in handling the unexpected config of "encrypt=false" whilst having the vault:secrets relation made. This crashes the charm, where it should handle it gracefully and put a note in the logs about not needing the relation.

The bug came about due to a change in charm-helpers which introduced the hvac requirement for the vault:secrets relation handling. This means that the affected charms need to have the python3-hvac (or equivalent for a virtualenv-ed environment) installed as default so that they don't crash.

Affected charms:

gnocchi
octavia

[Alex Kavanagh (ajkavanagh)]

The affected charms are not gnocchi and octavia. They are (or include):

ceph-osd
nova-compute
swift-storage

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/707704

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/707703
Committed: https://git.openstack.org/cgit/openstack/charm-swift-storage/commit/?id=91394ea9d64fba5cddd80614fd97ee891dd1f3e2
Submitter: Zuul
Branch: master

commit 91394ea9d64fba5cddd80614fd97ee891dd1f3e2
Author: Alex Kavanagh <email address hidden>
Date: Thu Feb 13 16:03:43 2020 +0000

    Ensure python3-hvac is installed for charms with encypt option

    The referenced bug is essentially: make vault:secrets relation to vault
    but keep the 'encrypt' option as False. In this case, the Context
    handling code in charm-helpers is expecting python3-hvac to be
    available, but it is only installed if the encrypt option is set to
    True. Hence the charm crashes. This resolves that crash.

    Note the related charm-helpers fix [1].

    [1]: https://github.com/juju/charm-helpers/pull/431

    Change-Id: I92773b7c1f48d456091062751e69581fabe4c5f3
    Closes-bug: #1862085

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/707705
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-osd/commit/?id=8b2e85e43532bb52797fd7f04f8d14db27a14f85
Submitter: Zuul
Branch: master

commit 8b2e85e43532bb52797fd7f04f8d14db27a14f85
Author: Alex Kavanagh <email address hidden>
Date: Thu Feb 13 16:07:28 2020 +0000

    Ensure python3-hvac is installed for charms with encypt option

    The referenced bug is essentially: make vault:secrets relation to vault
    but keep the 'encrypt' option as False. In this case, the Context
    handling code in charm-helpers is expecting python3-hvac to be
    available, but it is only installed if the encrypt option is set to
    True. Hence the charm crashes. This resolves that crash.

    Note the related charm-helpers fix [1].

    [1]: https://github.com/juju/charm-helpers/pull/431

    Change-Id: I9cb60a9340554c91668272b46f7c2dcf9f0ac2d1
    Closes-bug: #1862085

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/707704
Committed: https://git.openstack.org/cgit/openstack/charm-nova-compute/commit/?id=65d162aff282080397d02a37cdce2d1a670db227
Submitter: Zuul
Branch: master

commit 65d162aff282080397d02a37cdce2d1a670db227
Author: Alex Kavanagh <email address hidden>
Date: Thu Feb 13 16:06:38 2020 +0000

    Ensure python3-hvac is installed for charms with encypt option

    The referenced bug is essentially: make vault:secrets relation to vault
    but keep the 'encrypt' option as False. In this case, the Context
    handling code in charm-helpers is expecting python3-hvac to be
    available, but it is only installed if the encrypt option is set to
    True. Hence the charm crashes. This resolves that crash.

    Note the related charm-helpers fix [1].

    [1]: https://github.com/juju/charm-helpers/pull/431

    Change-Id: Ic02d4d4d3c3b423fa28cd171b126ed4a444fc646
    Closes-bug: #1862085