vault 1.7 token create failed, local node not active but active cluster node not found

[Alexander Balderson]

On a Jammy deployment on AWS with a single vault node running 1.7/stable charm. vault successfully ran operator init and unsealed, but creating the token failed with an error making the API request:

juju exec -m foundations-aws:kubernetes-aws -u vault/leader 'export VAULT_TOKEN=s.6wi8c8wGt50Jc1w0hNXDoudM && export VAULT_ADDR=http://127.0.0.1:8200 && /snap/bin/vault token create --ttl=5m'
2022-08-24-23:16:22 root ERROR [localhost] STDOUT follows:
b''
2022-08-24-23:16:22 root ERROR [localhost] STDERR follows:
Error creating token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/create
Code: 500. Errors:

* local node not active but active cluster node not found

There doesnt seem to be any helpful logging in the juju output or in the syslog about the failure.

[Nobuto Murata (nobuto)]

I can confirm this behavior. From what I saw, it looks like some moments are necessary to settle down the HA status of the vault cluster after unsealing it even with a single node cluster.

Here is an output from the unseal command and as we can see "n/a" is recorded for HA Cluster status and there is no cluster address either.

+ vault operator unseal Lp0QKU6ts/9Eydb/A3s3DUs5qcyzmxfpDI5M4jfmhY8=
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.8.8
Storage Type raft
Cluster Name vault-cluster-777ad2a5
Cluster ID 0afa6371-5941-493f-d258-d76389cdbecc
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address <none>
Raft Committed Index 24
Raft Applied Index 24

With the status above, `token create` can fail by missing active cluster node.

++ vault token create -ttl=10m -format json
Error creating token: Error making API request.

URL: POST http://192.168.151.108:8200/v1/auth/token/create
Code: 500. Errors:

* local node not active but active cluster node not found

Here is the expected status after some time for the record.

$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.8.8
Storage Type raft
Cluster Name vault-cluster-777ad2a5
Cluster ID 0afa6371-5941-493f-d258-d76389cdbecc
HA Enabled true
HA Cluster https://192.168.151.108:8201
HA Mode active
Active Since 2023-10-18T15:22:40.840977987Z
Raft Committed Index 1478
Raft Applied Index 1478

[Nobuto Murata (nobuto)]

A quick trick would be something like the following before executing `token create`.

until vault status -format json | jq -r --exit-status .leader_cluster_address; do
    sleep 1
done

[just after unsealing]
{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.8.8",
  "migration": false,
  "cluster_name": "vault-cluster-eb767fbb",
  "cluster_id": "3011c079-df2d-bd0d-dd3b-1e8c32a4ecf8",
  "recovery_seal": false,
  "storage_type": "raft",
  "ha_enabled": true,
  "active_time": "0001-01-01T00:00:00Z",
  "raft_committed_index": 24,
  "raft_applied_index": 24
}

[after settling down]
{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.8.8",
  "migration": false,
  "cluster_name": "vault-cluster-eb767fbb",
  "cluster_id": "3011c079-df2d-bd0d-dd3b-1e8c32a4ecf8",
  "recovery_seal": false,
  "storage_type": "raft",
  "ha_enabled": true,
  "is_self": true,
  "active_time": "2023-10-19T02:24:16.7102222Z",
  "leader_address": "http://192.168.151.110:8200",
  "leader_cluster_address": "https://192.168.151.110:8201",
  "raft_committed_index": 43,
  "raft_applied_index": 43
}