vault failed in shared-db-relation-changed hook KeyError: 'keys'

Bug #1792603 reported by Ashley Lai
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
vault-charm
Fix Released
High
Chris MacNaughton

Bug Description

xenial-queens

vault:
    charm: "cs:vault"
    options:
      totally-unsecure-auto-unlock: true

vault hook failed error from the log:

2018-09-14 10:54:08 DEBUG juju-log shared-db:230: Calling opportunistic_restart
2018-09-14 10:54:08 DEBUG shared-db-relation-changed inactive
2018-09-14 10:54:08 DEBUG juju-log shared-db:230: Safe to restart: True
2018-09-14 10:54:08 DEBUG juju-log shared-db:230: Restarting vault
2018-09-14 10:54:08 DEBUG shared-db-relation-changed active
2018-09-14 10:54:15 ERROR juju-log shared-db:230: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/__init__.py", line 73, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 382, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 358, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 180, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-vault-0/charm/reactive/vault_handlers.py", line 372, in file_change_auto_unlock_mode
    vault.prepare_vault()
  File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 243, in prepare_vault
    unseal_vault()
  File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 271, in unseal_vault
    keys = json.loads(hookenv.leader_get()['keys'])
KeyError: 'keys'
2018-09-14 10:54:15 DEBUG shared-db-relation-changed Traceback (most recent call last):
2018-09-14 10:54:15 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-vault-0/charm/hooks/shared-db-relation-changed", line 19, in <module>
2018-09-14 10:54:15 DEBUG shared-db-relation-changed main()
2018-09-14 10:54:15 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/__init__.py", line 73, in main
2018-09-14 10:54:15 DEBUG shared-db-relation-changed bus.dispatch(restricted=restricted_mode)
2018-09-14 10:54:15 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 382, in dispatch
2018-09-14 10:54:15 DEBUG shared-db-relation-changed _invoke(other_handlers)
2018-09-14 10:54:15 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 358, in _invoke
2018-09-14 10:54:15 DEBUG shared-db-relation-changed handler.invoke()
2018-09-14 10:54:15 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 180, in invoke
2018-09-14 10:54:15 DEBUG shared-db-relation-changed self._action(*args)
2018-09-14 10:54:15 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-vault-0/charm/reactive/vault_handlers.py", line 372, in file_change_auto_unlock_mode
2018-09-14 10:54:15 DEBUG shared-db-relation-changed vault.prepare_vault()
2018-09-14 10:54:15 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 243, in prepare_vault
2018-09-14 10:54:15 DEBUG shared-db-relation-changed unseal_vault()
2018-09-14 10:54:15 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 271, in unseal_vault
2018-09-14 10:54:15 DEBUG shared-db-relation-changed keys = json.loads(hookenv.leader_get()['keys'])
2018-09-14 10:54:15 DEBUG shared-db-relation-changed KeyError: 'keys'
2018-09-14 10:54:15 ERROR juju.worker.uniter.operation runhook.go:114 hook "shared-db-relation-changed" failed: exit status 1
2018-09-14 10:54:20 INFO juju-log shared-db:230: Reactive main running for hook shared-db-relation-changed

Revision history for this message
Ashley Lai (alai) wrote :
Revision history for this message
Ashley Lai (alai) wrote :
Revision history for this message
Ashley Lai (alai) wrote :
Revision history for this message
Ashley Lai (alai) wrote :

Please note that juju status shows vault units are active and OK. However CI is using "juju wait" and it sees that error before vault resolved itself.

Ryan Beisner (1chb1n)
Changed in vault-charm:
assignee: nobody → Chris MacNaughton (chris.macnaughton)
importance: Undecided → High
status: New → In Progress
Revision history for this message
Liam Young (gnuoy) wrote :

I think that we should remove the 'totally-unsecure-auto-unlock' option. It was introduced to help with automated tested but it isn't actually used by vaults functional tests. It is probably more appropriate for automated testing to run the post deployment actions as this more closely mirrors what will be done in the wild.

Changed in vault-charm:
status: In Progress → Fix Committed
Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

Thanks for fixing this. What's the timeframe for this being backported to stable?

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

@jason-hobbs can you describe in a bit more depth why the testing is using a mode that is explicitly recommended against for testing? I'm assuming that it is not being deployed with that configuration option enabled to customer deployments and would much prefer that we get QA testing aligned with field deployments and upstream (openstack-charms) testing rather than using a configuration option that we'd like to remove anyway.

Revision history for this message
Ashley Lai (alai) wrote :

Sure we can remove the 'totally-unsecure-auto-unlock' option.

David Ames (thedac)
Changed in vault-charm:
milestone: none → 19.04
Revision history for this message
James Page (james-page) wrote :

Marking fix released as this was part of the last charm release.

Changed in vault-charm:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.