Bug #1886424 “One barbican-vault unit is failing with “wrapping ...” : Bugs : OpenStack Barbican-Vault Charm

Revision history for this message

Alexander Litvinov (alitvinov) wrote on 2020-07-06:

#1

As suggested in the previous bug, running the refresh action below didn't help barbican-vault units..

$ juju run-action --wait vault/1 refresh-secrets
unit-vault-1:
UnitId: vault/1
status: completed

Even after juju resolved barbican-vault/5 and executing, units end up in error state.

Dmitrii Shcherbakov (dmitriis) on 2020-07-06

Changed in charm-barbican-vault:
status:	New → Triaged
importance:	Undecided → High

Alexander Litvinov (alitvinov) on 2020-07-06

description:

updated

Revision history for this message

Dmitrii Shcherbakov (dmitriis) wrote on 2020-07-06:

#2

Download full text (7.9 KiB)

A PDB trace of the request done at the failing unit (the request looks OK from the perspective of this doc https://www.vaultproject.io/api-docs/system/wrapping-unwrap#parameters but the token is likely the issue):

root@juju-b68657-23-lxd-10:/var/lib/juju/agents/unit-barbican-vault-5/charm# ./hooks/secrets-storage-relation-joined
lib/charm/vault_utils.py:32: DeprecationWarning: Call to deprecated function '_post'. This method will be removed in version '0.8.0' Please use the 'post' method on the 'hvac.adapters' class moving forward.
response = client._post('/v1/sys/wrapping/unwrap')
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(109)post()
-> return self.request('post', url, **kwargs)
(Pdb) url
'/v1/sys/wrapping/unwrap'
(Pdb) kwargs
{}

(Pdb) self.request
<bound method RawAdapter.request of <hvac.adapters.RawAdapter object at 0x7f79416a3c88>>
(Pdb) s
--Call--
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(248)request()
-> def request(self, method, url, headers=None, raise_exception=True, **kwargs):
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(266)request()
-> while '//' in url:
(Pdb) l
261 :param kwargs: Additional keyword arguments to include in the requests call.
262 :type kwargs: dict
263 :return: The response of the request.
264 :rtype: requests.Response
265 """
266 -> while '//' in url:
267 # Vault CLI treats a double forward slash ('//') as a single forward slash for a given path.
268 # To avoid issues with the requests module's redirection logic, we perform the same translation here.
269 url = url.replace('//', '/')
270
271 url = self.urljoin(self.base_uri, url)
(Pdb) url
'/v1/sys/wrapping/unwrap'
(Pdb) method
'post'
(Pdb) headers
(Pdb) kwargs
{}

(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(271)request()
-> url = self.urljoin(self.base_uri, url)
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(273)request()
-> if not headers:
(Pdb) url
'http://vault.<redacted-hostname>:8200/v1/sys/wrapping/unwrap'

-> if self.token:
(Pdb) l
271 url = self.urljoin(self.base_uri, url)
272
273 if not headers:
274 headers = {}
275
276 -> if self.token:
277 headers['X-Vault-Token'] = self.token
278
279 if self.namespace:
280 headers['X-Vault-Namespace'] = self.namespace
281
(Pdb) self.token
's.QYHb0vo4v2a6RpnwaOLNlvps'

(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(277)request()
-> headers['X-Vault-Token'] = self.token
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(279)request()
-> if self.namespace:
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(282)request()
-> wrap_ttl = kwargs.pop('wrap_ttl', No...

A PDB trace of the request done at the failing unit (the request looks OK from the perspective of this doc https://www.vaultproject.io/api-docs/system/wrapping-unwrap#parameters but the token is likely the issue):

root@juju-b68657-23-lxd-10:/var/lib/juju/agents/unit-barbican-vault-5/charm# ./hooks/secrets-storage-relation-joined
lib/charm/vault_utils.py:32: DeprecationWarning: Call to deprecated function '_post'. This method will be removed in version '0.8.0' Please use the 'post' method on the 'hvac.adapters' class moving forward.
  response = client._post('/v1/sys/wrapping/unwrap')
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(109)post()
-> return self.request('post', url, **kwargs)
(Pdb) url
'/v1/sys/wrapping/unwrap'
(Pdb) kwargs
{}

(Pdb) self.request
<bound method RawAdapter.request of <hvac.adapters.RawAdapter object at 0x7f79416a3c88>>
(Pdb) s
--Call--
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(248)request()
-> def request(self, method, url, headers=None, raise_exception=True, **kwargs):
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(266)request()
-> while '//' in url:
(Pdb) l
261             :param kwargs: Additional keyword arguments to include in the requests call.
262             :type kwargs: dict
263             :return: The response of the request.
264             :rtype: requests.Response
265             """
266  ->         while '//' in url:
267                 # Vault CLI treats a double forward slash ('//') as a single forward slash for a given path.
268                 # To avoid issues with the requests module's redirection logic, we perform the same translation here.
269                 url = url.replace('//', '/')
270
271             url = self.urljoin(self.base_uri, url)
(Pdb) url
'/v1/sys/wrapping/unwrap'
(Pdb) method
'post'
(Pdb) headers
(Pdb) kwargs
{}

(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(271)request()
-> url = self.urljoin(self.base_uri, url)
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(273)request()
-> if not headers:
(Pdb) url
'http://vault.<redacted-hostname>:8200/v1/sys/wrapping/unwrap'

-> if self.token:
(Pdb) l
271             url = self.urljoin(self.base_uri, url)
272
273             if not headers:
274                 headers = {}
275
276  ->         if self.token:
277                 headers['X-Vault-Token'] = self.token
278
279             if self.namespace:
280                 headers['X-Vault-Namespace'] = self.namespace
281
(Pdb) self.token
's.QYHb0vo4v2a6RpnwaOLNlvps'

(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(277)request()
-> headers['X-Vault-Token'] = self.token
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(279)request()
-> if self.namespace:
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(282)request()
-> wrap_ttl = kwargs.pop('wrap_ttl', None)
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(283)request()
-> if wrap_ttl:
(Pdb) wrap_ttl
(Pdb)

(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(286)request()
-> _kwargs = self._kwargs.copy()
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(287)request()
-> _kwargs.update(kwargs)
(Pdb) _kwargs
{'cert': None, 'verify': '/etc/ssl/certs/ca-certificates.crt', 'timeout': 30, 'proxies': None}
(Pdb) _kwargs
{'cert': None, 'verify': '/etc/ssl/certs/ca-certificates.crt', 'timeout': 30, 'proxies': None}

(Pdb) l
282             wrap_ttl = kwargs.pop('wrap_ttl', None)
283             if wrap_ttl:
284                 headers['X-Vault-Wrap-TTL'] = str(wrap_ttl)
285
286             _kwargs = self._kwargs.copy()
287  ->         _kwargs.update(kwargs)
288
289             response = self.session.request(
290                 method=method,
291                 url=url,
292                 headers=headers,
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(289)request()
-> response = self.session.request(
(Pdb) method
'post'
(Pdb) url
'http://vault.<redacted-hostname>:8200/v1/sys/wrapping/unwrap'
(Pdb) headers
{'X-Vault-Token': 's.QYHb0vo4v2a6RpnwaOLNlvps'}

(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/hvac/adapters.py(290)request()
-> method=method,
(Pdb) l
285
286             _kwargs = self._kwargs.copy()
287             _kwargs.update(kwargs)
288
289             response = self.session.request(
290  ->             method=method,
291                 url=url,
292                 headers=headers,
293                 allow_redirects=self.allow_redirects,
294                 **_kwargs
295             )

(Pdb) method
'post'
(Pdb) url
'http://vault.<redacted-hostname>:8200/v1/sys/wrapping/unwrap'
(Pdb) headers
{'X-Vault-Token': 's.QYHb0vo4v2a6RpnwaOLNlvps'}
(Pdb) self.allow_redirects
True
(Pdb) _kwargs
{'cert': None, 'verify': '/etc/ssl/certs/ca-certificates.crt', 'timeout': 30, 'proxies': None}

---------------------------

> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/requests/adapters.py(440)send()
-> method=request.method,
(Pdb) l
435                 timeout = TimeoutSauce(connect=timeout, read=timeout)
436
437             try:
438                 if not chunked:
439                     resp = conn.urlopen(
440  ->                     method=request.method,
441                         url=url,
442                         body=request.body,
443                         headers=request.headers,
444                         redirect=False,
445                         assert_same_host=False,
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/requests/adapters.py(441)send()
-> url=url,
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/requests/adapters.py(442)send()
-> body=request.body,
(Pdb) n
> /var/lib/juju/agents/unit-barbican-vault-5/.venv/lib/python3.6/site-packages/requests/adapters.py(443)send()
-> headers=request.headers,
(Pdb) request.body
(Pdb) request.headers
{'User-Agent': 'python-requests/2.23.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'X-Vault-Token': 's.QYHb0vo4v2a6RpnwaOLNlvps', 'Content-Length': '0'}

...

(Pdb) response
<Response [400]>
(Pdb) response.__dict__
{'_content': False, '_content_consumed': False, '_next': None, 'status_code': 400, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'Date': 'Mon, 06 Jul 2020 11:33:37 GMT', 'Content-Length': '61'}, 'raw': <urllib3.response.HTTPResponse object at 0x7fd81b40de48>, 'url': None, 'encoding': None, 'history': [], 'reason': 'Bad Request', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(0), 'request': None}

(Pdb) response.raw
<urllib3.response.HTTPResponse object at 0x7fd81b40de48>
(Pdb) response.raw.__dict__
{'headers': HTTPHeaderDict({'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'Date': 'Mon, 06 Jul 2020 11:33:37 GMT', 'Content-Length': '61'}), 'status': 400, 'version': 11, 'reason': 'Bad Request', 'strict': 0, 'decode_content': False, 'retries': Retry(total=0, connect=None, read=False, redirect=None, status=None), 'enforce_content_length': False, 'auto_close': True, '_decoder': None, '_body': None, '_fp': <http.client.HTTPResponse object at 0x7fd81b40dd68>, '_original_response': <http.client.HTTPResponse object at 0x7fd81b40dd68>, '_fp_bytes_read': 0, 'msg': None, '_request_url': None, '_pool': <urllib3.connectionpool.HTTPConnectionPool object at 0x7fd81b465eb8>, '_connection': <urllib3.connection.HTTPConnection object at 0x7fd81b40dc18>, 'chunked': False, 'chunk_left': None, 'length_remaining': 61}

Revision history for this message

Alexander Litvinov (alitvinov) wrote on 2020-07-06:

#3

sosreport from failed unit:
https://private-fileshare.canonical.com/~alitvinov/random/sosreport-barbican-failed.tar.xz

Revision history for this message

Alexander Litvinov (alitvinov) wrote on 2020-07-06:

#4

juju-crashdump:
https://private-fileshare.canonical.com/~alitvinov/random/juju-crashdump-60f81344-b8fc-4e69-a344-f2bc5af602fb.tar.xz

Revision history for this message

Alexander Litvinov (alitvinov) wrote on 2020-07-06:

#5

Subscribing ~field-critical as deployment is blocked and workaround with refresh-secrets didn't help

Revision history for this message

Dmitrii Shcherbakov (dmitriis) wrote on 2020-07-06:

#6

The response from Vault we see happens right after the wrapping token validation (if it fails):
https://github.com/hashicorp/vault/blob/v1.1.1/http/handler.go#L297-L303
valid, err := core.ValidateWrappingToken(ctx, req)
if err != nil {
return errwrap.Wrapf("error validating wrapping token: {{err}}", err)
}
if !valid {
return fmt.Errorf("wrapping token is not valid or does not exist")
}

It is possible to set TTL for wrapping tokens:
https://www.vaultproject.io/docs/concepts/response-wrapping#response-wrapping-token-creation

The TTL for one-time tokens seems to be 1h as configured in charm-vault during the secret creation:

https://opendev.org/openstack/charm-vault/src/branch/stable/20.05/src/reactive/vault_handlers.py#L491-L499
        if new_role or refresh_secrets:
            wrapped_secret = vault.generate_role_secret_id(
                client,
                name=approle_name,
                cidr=cidr
            )
            secrets.set_role_id(unit=unit,
                                role_id=approle_id,
                                token=wrapped_secret)

Note 1h here:
https://opendev.org/openstack/charm-vault/src/branch/stable/20.05/src/lib/charm/vault.py#L395-L408
def generate_role_secret_id(client, name, cidr):
# ...
    response = client.write('auth/approle/role/{}/secret-id'.format(name),
                            wrap_ttl='1h', cidr_list=cidr)
    return response['wrap_info']['token']

https://hvac.readthedocs.io/en/stable/source/hvac_v1.html#hvac.v1.Client.create_role_secret_id

The wrapping token is then passed through the vault-kv interface to barbican-vault:

https://github.com/openstack-charmers/charm-interface-vault-kv/blob/5e71b61c1ddb6ecaade2b6675a5d8cf26655d7b0/requires.py#L75-L88
    @property
    def all_unit_tokens(self):
        """Retrieve the one-shot token(s) for secret_id retrieval for
        all application units or empty list.
        :returns token: Vault one-shot token for secret_id response
        :rtype token: str"""
        token_key = '{}_token'.format(hookenv.local_unit())
        tokens = set()
        for relation in self.relations:
            for unit in relation.units:
                token = unit.received.get(token_key)
                if token:
                    tokens.add(token)

https://github.com/openstack-charmers/charm-interface-vault-kv/blob/5e71b61c1ddb6ecaade2b6675a5d8cf26655d7b0/provides.py#L67-L74
    def set_role_id(self, unit, role_id, token):
        """ Set the AppRole ID and token for out-of-band Secret ID retrieval
        for a specific remote unit """
        # for cmr we will need to the other end to provide their unit name
        # expicitly.
        unit_name = unit.received.get('unit_name') or unit.unit_name
        unit.relation.to_publish['{}_role_id'.format(unit_name)] = role_id
        unit.relation.to_publish['{}_token'.format(unit_name)] = token

The response from Vault we see happens right after the wrapping token validation (if it fails):
https://github.com/hashicorp/vault/blob/v1.1.1/http/handler.go#L297-L303
	valid, err := core.ValidateWrappingToken(ctx, req)
	if err != nil {
		return errwrap.Wrapf("error validating wrapping token: {{err}}", err)
	}
	if !valid {
		return fmt.Errorf("wrapping token is not valid or does not exist")
	}

It is possible to set TTL for wrapping tokens:
https://www.vaultproject.io/docs/concepts/response-wrapping#response-wrapping-token-creation

The TTL for one-time tokens seems to be 1h as configured in charm-vault during the secret creation:

https://opendev.org/openstack/charm-vault/src/branch/stable/20.05/src/reactive/vault_handlers.py#L491-L499
        if new_role or refresh_secrets:
            wrapped_secret = vault.generate_role_secret_id(
                client,
                name=approle_name,
                cidr=cidr
            )
            secrets.set_role_id(unit=unit,
                                role_id=approle_id,
                                token=wrapped_secret)

Note 1h here:
https://opendev.org/openstack/charm-vault/src/branch/stable/20.05/src/lib/charm/vault.py#L395-L408
def generate_role_secret_id(client, name, cidr):
# ...
    response = client.write('auth/approle/role/{}/secret-id'.format(name),
                            wrap_ttl='1h', cidr_list=cidr)
    return response['wrap_info']['token']

https://hvac.readthedocs.io/en/stable/source/hvac_v1.html#hvac.v1.Client.create_role_secret_id

The wrapping token is then passed through the vault-kv interface to barbican-vault:

https://github.com/openstack-charmers/charm-interface-vault-kv/blob/5e71b61c1ddb6ecaade2b6675a5d8cf26655d7b0/requires.py#L75-L88
    @property
    def all_unit_tokens(self):
        """Retrieve the one-shot token(s) for secret_id retrieval for
        all application units or empty list.
        :returns token: Vault one-shot token for secret_id response
        :rtype token: str"""
        token_key = '{}_token'.format(hookenv.local_unit())
        tokens = set()
        for relation in self.relations:
            for unit in relation.units:
                token = unit.received.get(token_key)
                if token:
                    tokens.add(token)

https://github.com/openstack-charmers/charm-interface-vault-kv/blob/5e71b61c1ddb6ecaade2b6675a5d8cf26655d7b0/provides.py#L67-L74
    def set_role_id(self, unit, role_id, token):
        """ Set the AppRole ID and token for out-of-band Secret ID retrieval
        for a specific remote unit """
        # for cmr we will need to the other end to provide their unit name
        # expicitly.
        unit_name = unit.received.get('unit_name') or unit.unit_name
        unit.relation.to_publish['{}_role_id'.format(unit_name)] = role_id
        unit.relation.to_publish['{}_token'.format(unit_name)] = token

Revision history for this message

Dmitrii Shcherbakov (dmitriis) wrote on 2020-07-06:

#7

HA looks OK:

* 1 unit is active;
* 2 units are standby;
* 1/2 standbys holds a VIP and redirects client requests to the active node. (https://www.vaultproject.io/docs/concepts/ha#server-to-server-communication)

As far as I can tell, the problem is somewhere else.

ubuntu@vault-3:~$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.1.1
Cluster Name vault-cluster-df1658d2
Cluster ID 5a141718-4814-507f-0e51-0cfc6ac02905
HA Enabled true
HA Cluster https://10.201.12.7:8201
HA Mode active

ubuntu@vault-2:~$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.1.1
Cluster Name vault-cluster-df1658d2
Cluster ID 5a141718-4814-507f-0e51-0cfc6ac02905
HA Enabled true
HA Cluster https://10.201.12.7:8201
HA Mode standby
Active Node Address http://10.201.12.7:8200

ubuntu@vault-1:~$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.1.1
Cluster Name vault-cluster-df1658d2
Cluster ID 5a141718-4814-507f-0e51-0cfc6ac02905
HA Enabled true
HA Cluster https://10.201.12.7:8201
HA Mode standby
Active Node Address http://10.201.12.7:8200

ubuntu@vault-1:~$ ip -4 -o a s
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
2: eth0 inet 10.201.11.170/24 brd 10.201.11.255 scope global eth0\ valid_lft forever preferred_lft forever
3: eth1 inet 10.201.12.28/24 brd 10.201.12.255 scope global eth1\ valid_lft forever preferred_lft forever
3: eth1 inet 10.201.12.43/24 brd 10.201.12.255 scope global secondary eth1\ valid_lft forever preferred_lft forever

ubuntu@vault-1:~$ sudo crm status
Stack: corosync
Current DC: vault-3 (version 1.1.18-2b07d5c5a9) - partition with quorum
Last updated: Mon Jul 6 17:35:48 2020
Last change: Fri Jul 3 12:28:21 2020 by hacluster via crmd on vault-3

3 nodes configured
1 resource configured

Online: [ vault-1 vault-2 vault-3 ]

Full list of resources:

Resource Group: grp_vault-ext_vips
res_vault-ext_b547a20_vip (ocf::heartbeat:IPaddr2): Started vault-1

HA looks OK:

* 1 unit is active;
* 2 units are standby;
* 1/2 standbys holds a VIP and redirects client requests to the active node. (https://www.vaultproject.io/docs/concepts/ha#server-to-server-communication)

As far as I can tell, the problem is somewhere else.

ubuntu@vault-3:~$ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.1.1
Cluster Name    vault-cluster-df1658d2
Cluster ID      5a141718-4814-507f-0e51-0cfc6ac02905
HA Enabled      true
HA Cluster      https://10.201.12.7:8201
HA Mode         active

ubuntu@vault-2:~$ vault status
Key                    Value
---                    -----
Seal Type              shamir
Initialized            true
Sealed                 false
Total Shares           5
Threshold              3
Version                1.1.1
Cluster Name           vault-cluster-df1658d2
Cluster ID             5a141718-4814-507f-0e51-0cfc6ac02905
HA Enabled             true
HA Cluster             https://10.201.12.7:8201
HA Mode                standby
Active Node Address    http://10.201.12.7:8200

ubuntu@vault-1:~$ vault status
Key                    Value
---                    -----
Seal Type              shamir
Initialized            true
Sealed                 false
Total Shares           5
Threshold              3
Version                1.1.1
Cluster Name           vault-cluster-df1658d2
Cluster ID             5a141718-4814-507f-0e51-0cfc6ac02905
HA Enabled             true
HA Cluster             https://10.201.12.7:8201
HA Mode                standby
Active Node Address    http://10.201.12.7:8200

ubuntu@vault-1:~$ ip -4 -o a s
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
2: eth0    inet 10.201.11.170/24 brd 10.201.11.255 scope global eth0\       valid_lft forever preferred_lft forever
3: eth1    inet 10.201.12.28/24 brd 10.201.12.255 scope global eth1\       valid_lft forever preferred_lft forever
3: eth1    inet 10.201.12.43/24 brd 10.201.12.255 scope global secondary eth1\       valid_lft forever preferred_lft forever

ubuntu@vault-1:~$ sudo crm status
Stack: corosync
Current DC: vault-3 (version 1.1.18-2b07d5c5a9) - partition with quorum
Last updated: Mon Jul  6 17:35:48 2020
Last change: Fri Jul  3 12:28:21 2020 by hacluster via crmd on vault-3

3 nodes configured
1 resource configured

Online: [ vault-1 vault-2 vault-3 ]

Full list of resources:

Resource Group: grp_vault-ext_vips
     res_vault-ext_b547a20_vip  (ocf::heartbeat:IPaddr2):       Started vault-1

Revision history for this message

Dmitrii Shcherbakov (dmitriis) wrote on 2020-07-06:

#8

An observation:

adding new barbican-vault units results in the same failure during secrets-storage-relation-joined (even though new wrapping tokens are generated for them by charm-vault)

Revision history for this message

James Page (james-page) wrote on 2020-07-08:

#9

For further context the token has a 1h ttl and is limited to access from a specific IP address.

Revision history for this message

James Page (james-page) wrote on 2020-07-08:

#10

This worries me:

  barbican-vault:
    charm: cs:barbican-vault-15
...
    endpoint-bindings:
      "": alpha
      certificates: alpha
      juju-info: alpha
      secrets: alpha
      secrets-storage: alpha

  vault:
    charm: cs:vault-39
...
    endpoint-bindings:
      "": internal-space
      access: internal-space
      certificates: internal-space
      cluster: internal-space
      db: internal-space
      etcd: internal-space
      external: internal-space
      ha: internal-space
      nrpe-external-master: internal-space
      secrets: internal-space
      shared-db: internal-space

the 'alpha' endpoint bindings on the barbican-vault charm mean no explicit endpoint binding is provided and Juju has somewhat EUNDEFINED behaviour as to what network-get returns on any given unit in this case.

Please ensure "": internal-space is use in the bundle.

Revision history for this message

Alexander Litvinov (alitvinov) wrote on 2020-07-08:

#11

After deploying with updated bindings all units are green.

juju deploy cs:barbican-vault --bind "oam-space secrets-storage:internal-space secrets:internal-space certificates:internal-space"

Thank you James end Dmitrii for helping with this issue

Revision history for this message

Dmitrii Shcherbakov (dmitriis) wrote on 2020-07-08:

#12

Glad it helped.

Thanks to James for pointing the token IP ACL bit out.

Changed in charm-barbican-vault:
status:	Triaged → Invalid

OpenStack Barbican-Vault Charm

One barbican-vault unit is failing with "wrapping token is not valid or does not exist"

Bug Description

Other bug subscribers

Remote bug watches