vault-charm

Vault unit failes during secrets-relations-changed

Bug #1921768 reported by Alexander Balderson on 2021-03-29

This bug affects 5 people

Affects		Status	Importance	Assigned to	Milestone
	vault-charm	Confirmed	Undecided	Unassigned

Bug Description

One vault unit (vault 2) failed during the secrets-relation-joined hook with the following traceback:
2021-03-29 15:09:45 WARNING secrets-relation-joined Traceback (most recent call last):
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/charm/hooks/secrets-relation-joined", line 22, in <module>
2021-03-29 15:09:45 WARNING secrets-relation-joined main()
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 84, in main
2021-03-29 15:09:45 WARNING secrets-relation-joined hookenv._run_atexit()
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/charmhelpers/core/hookenv.py", line 1343, in _run_atexit
2021-03-29 15:09:45 WARNING secrets-relation-joined callback(*args, **kwargs)
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/charm/reactive/vault_handlers.py", line 758, in _assess_status
2021-03-29 15:09:45 WARNING secrets-relation-joined if not client_approle_authorized():
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/charm/reactive/vault_handlers.py", line 788, in client_approle_authorized
2021-03-29 15:09:45 WARNING secrets-relation-joined vault.get_local_client()
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 329, in wrapped_f
2021-03-29 15:09:45 WARNING secrets-relation-joined return self.call(f, *args, **kw)
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 409, in call
2021-03-29 15:09:45 WARNING secrets-relation-joined do = self.iter(retry_state=retry_state)
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 356, in iter
2021-03-29 15:09:45 WARNING secrets-relation-joined return fut.result()
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
2021-03-29 15:09:45 WARNING secrets-relation-joined return self.__get_result()
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
2021-03-29 15:09:45 WARNING secrets-relation-joined raise self._exception
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 412, in call
2021-03-29 15:09:45 WARNING secrets-relation-joined result = fn(*args, **kwargs)
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/charm/lib/charm/vault.py", line 254, in get_local_client
2021-03-29 15:09:45 WARNING secrets-relation-joined client.auth_approle(app_role_id)
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 2072, in auth_approle
2021-03-29 15:09:45 WARNING secrets-relation-joined return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token)
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 1726, in auth
2021-03-29 15:09:45 WARNING secrets-relation-joined return self._adapter.auth(
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 159, in auth
2021-03-29 15:09:45 WARNING secrets-relation-joined response = self.post(url, **kwargs).json()
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 103, in post
2021-03-29 15:09:45 WARNING secrets-relation-joined return self.request('post', url, **kwargs)
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 233, in request
2021-03-29 15:09:45 WARNING secrets-relation-joined utils.raise_for_error(response.status_code, text, errors=errors)
2021-03-29 15:09:45 WARNING secrets-relation-joined File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/utils.py", line 29, in raise_for_error
2021-03-29 15:09:45 WARNING secrets-relation-joined raise exceptions.InvalidRequest(message, errors=errors)
2021-03-29 15:09:45 WARNING secrets-relation-joined hvac.exceptions.InvalidRequest: missing client token

The test was running Usuri on bionic with OVS.

the test run can be found at https://solutions.qa.canonical.com/testruns/testRun/face2065-73d6-4840-8766-125d0c2db7e2

Tags:

Revision history for this message

Michał Ajduk (majduk) wrote on 2021-04-28:

Reproducer bundle Edit (1.5 KiB, text/plain)

Download full text (7.7 KiB)

I was hit by a very similiar bug. Reproducer bundle attached.

Charm version: cs:vault-44

After vault initialization, unsealing and charm authorisation I get following vault status:
Model Controller Cloud/Region Version SLA Timestamp
lma foundation-openstack openstack_cloud/us-east 2.8.10 unsupported 07:21:09Z

Unit Workload Agent Machine Public address Ports Message
vault/0* active idle 3 10.24.100.190 8200/tcp Unit is ready (active: true, mlock: enabled)
  hacluster-vault/0* active idle 10.24.100.190 Unit is ready and clustered
  vault-mysql-router/0* active idle 10.24.100.190 Unit is ready
vault/1 error idle 4 10.24.100.179 8200/tcp hook failed: "leader-settings-changed"
  hacluster-vault/2 active idle 10.24.100.179 Unit is ready and clustered
  vault-mysql-router/2 active idle 10.24.100.179 Unit is ready
vault/2 error idle 5 10.24.100.26 8200/tcp hook failed: "leader-settings-changed"
  hacluster-vault/1 active idle 10.24.100.26 Unit is ready and clustered
  vault-mysql-router/1 active idle 10.24.100.26 Unit is ready

Machine State DNS Inst id Series AZ Message
3 started 10.24.100.190 934b66b1-aa29-49ed-8f5b-12d31f9bef28 focal AZ1 ACTIVE
4 started 10.24.100.179 d1e0f50c-5e4f-402b-a0e2-17de98eec6d1 focal AZ2 ACTIVE
5 started 10.24.100.26 7c099246-4b3a-4ffc-a9a1-b52eaab24592 focal AZ3 ACTIVE

I was hit by a very similiar bug. Reproducer bundle attached.

Charm version: cs:vault-44

After vault initialization, unsealing and charm authorisation I get following vault status:
Model  Controller            Cloud/Region             Version  SLA          Timestamp
lma    foundation-openstack  openstack_cloud/us-east  2.8.10   unsupported  07:21:09Z

App                 Version  Status  Scale  Charm         Store       Rev  OS      Notes
hacluster-vault              active      3  hacluster     jujucharms   74  ubuntu  
vault               1.5.4    error       3  vault         jujucharms   44  ubuntu  
vault-mysql-router  8.0.23   active      3  mysql-router  jujucharms    6  ubuntu

Unit                     Workload  Agent  Machine  Public address  Ports     Message
vault/0*                 active    idle   3        10.24.100.190   8200/tcp  Unit is ready (active: true, mlock: enabled)
  hacluster-vault/0*     active    idle            10.24.100.190             Unit is ready and clustered
  vault-mysql-router/0*  active    idle            10.24.100.190             Unit is ready
vault/1                  error     idle   4        10.24.100.179   8200/tcp  hook failed: "leader-settings-changed"
  hacluster-vault/2      active    idle            10.24.100.179             Unit is ready and clustered
  vault-mysql-router/2   active    idle            10.24.100.179             Unit is ready
vault/2                  error     idle   5        10.24.100.26    8200/tcp  hook failed: "leader-settings-changed"
  hacluster-vault/1      active    idle            10.24.100.26              Unit is ready and clustered
  vault-mysql-router/1   active    idle            10.24.100.26              Unit is ready

Machine  State    DNS            Inst id                               Series  AZ   Message
3        started  10.24.100.190  934b66b1-aa29-49ed-8f5b-12d31f9bef28  focal   AZ1  ACTIVE
4        started  10.24.100.179  d1e0f50c-5e4f-402b-a0e2-17de98eec6d1  focal   AZ2  ACTIVE
5        started  10.24.100.26   7c099246-4b3a-4ffc-a9a1-b52eaab24592  focal   AZ3  ACTIVE

Relevant log section:
2021-04-28 07:08:50 WARNING leader-settings-changed All snaps up to date.
2021-04-28 07:08:50 INFO juju-log Initializing Leadership Layer (is follower)
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:253:configure_vault_mysql
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:294:mysql_setup
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:325:database_not_ready
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:415:cluster_connected
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:598:prime_assess_status
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: hooks/relations/tls-certificates/provides.py:63:broken:certificates
2021-04-28 07:08:51 WARNING leader-settings-changed Traceback (most recent call last):
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/charm/hooks/leader-settings-changed", line 22, in <module>
2021-04-28 07:08:51 WARNING leader-settings-changed     main()
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 84, 
in main
2021-04-28 07:08:51 WARNING leader-settings-changed     hookenv._run_atexit()
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/charmhelpers/core/hookenv.py", line 135
4, in _run_atexit
2021-04-28 07:08:51 WARNING leader-settings-changed     callback(*args, **kwargs)
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/charm/reactive/vault_handlers.py", line 759, in _assess_status
2021-04-28 07:08:51 WARNING leader-settings-changed     if not client_approle_authorized():
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/charm/reactive/vault_handlers.py", line 789, in client_approle_authorized2021-04-28 07:08:51 WARNING leader-settings-changed     vault.get_local_client()
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 333, in wrapped_f
2021-04-28 07:08:51 WARNING leader-settings-changed     return self(f, *args, **kw)
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 423, in __c
all__
2021-04-28 07:08:51 WARNING leader-settings-changed     do = self.iter(retry_state=retry_state)
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 360, in iter
2021-04-28 07:08:51 WARNING leader-settings-changed     return fut.result()
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
2021-04-28 07:08:51 WARNING leader-settings-changed     return self.__get_result()
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
2021-04-28 07:08:51 WARNING leader-settings-changed     raise self._exception
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 426, in __c
all__
2021-04-28 07:08:51 WARNING leader-settings-changed     result = fn(*args, **kwargs)
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/charm/lib/charm/vault.py", line 254, in get_local_client
2021-04-28 07:08:51 WARNING leader-settings-changed     client.auth_approle(app_role_id)
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 2072, in aut
h_approle
2021-04-28 07:08:51 WARNING leader-settings-changed     return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token)
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 1726, in auth
2021-04-28 07:08:51 WARNING leader-settings-changed     return self._adapter.auth(
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 159, in auth
2021-04-28 07:08:51 WARNING leader-settings-changed     response = self.post(url, **kwargs).json()
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 103, in post
2021-04-28 07:08:51 WARNING leader-settings-changed     return self.request('post', url, **kwargs)
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 233, in request
2021-04-28 07:08:51 WARNING leader-settings-changed     utils.raise_for_error(response.status_code, text, errors=errors)
2021-04-28 07:08:51 WARNING leader-settings-changed   File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/utils.py", line 29, in raise_for_error
2021-04-28 07:08:51 WARNING leader-settings-changed     raise exceptions.InvalidRequest(message, errors=errors)
2021-04-28 07:08:51 WARNING leader-settings-changed hvac.exceptions.InvalidRequest: missing client token
2021-04-28 07:08:52 ERROR juju.worker.uniter.operation runhook.go:139 hook "leader-settings-changed" (via explicit, bespoke hook script) failed: exit status 1

Haw Loeung (hloeung) on 2022-06-14