vault stops answering barbican-vault's token request

[Adam Dyess]

Vault can find its way into situations where barbican's CIDR/secrets_id/role_id tripplet results in an "unauthorized through CIDR restrictions".

This leaves the barbican unit unable to answer requests for secrets. The logs in barbican-api.log leave a lot to be desired [1]. Essentially, there's nothing catching bad responses for a token from vault. See this [2]

To debug the issue, we had to issue our own curl from the barbican unit:

cat /etc/barbican/barbican.conf | grep vault_plugin] -A 5
   secret=<secret>
   role=<role>
vault_url=<vault_url>
curl -vv -XPOST $vault_url/v1/auth/approle/login -d '{"secret_id": "'$secret'", "role_id": "'$role'"}' -H 'Content-Type: application/json'

Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 10.191.5.28...
* TCP_NODELAY set
* Connected to 10.191.5.28 (10.191.5.28) port 8200 (#0)
> POST /v1/auth/approle/login HTTP/1.1
> Host: 10.191.5.28:8200
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 104
>
* upload completely sent off: 104 out of 104 bytes
< HTTP/1.1 400 Bad Request
< Cache-Control: no-store
< Content-Type: application/json
< Date: Wed, 07 Apr 2021 20:27:47 GMT
< Content-Length: 119
<
{"errors":["source address \"10.191.5.91\" unauthorized through CIDR restrictions on the secret ID: \u003cnil\u003e"]}

As you can see, vault is rejecting the barbican unit. The only workaround was to deploy a new barbican unit and remove the old one

```
  juju config hacluster-barbican cluster_count=4
  juju add-unit barbican --to lxd:<bad-unit-metal>
  # wait for stable

  juju remove-unit barbican/<bad-unit>
  juju config hacluster-barbican cluster_count=3
  # wait for stable
```

[1] https://pastebin.canonical.com/p/tZ3SY3kDp5/
[2] https://github.com/openstack/castellan/blob/master/castellan/key_manager/vault_key_manager.py#L169