Comment 2 for bug 2017514

Tested to add a third unit today again. Vault stuck in "Vault needs to be initialized".
Executing changes:
- set application options for vault-hacluster
- add unit vault/5 to 2/lxd/0

vault/5 blocked executing 2/lxd/49 10.105.121.85 8200/tcp Vault needs to be initialized
  vault-hacluster/5 active idle 10.105.121.85 Unit is ready and clustered
  vault-mysql-router/5 active idle 10.105.121.85 Unit is ready

debug-log shows "Failed to join raft cluster: HTTPConnectionPool(host='127.0.0.1', port=8220): Read timed out":
...
unit-vault-5: 22:09:13 INFO unit.vault/5.juju-log certificates:439: Invoking reactive handler: reactive/vault_handlers.py:474:cluster_connected
unit-vault-5: 22:09:13 INFO unit.vault/5.juju-log certificates:439: Invoking reactive handler: reactive/vault_handlers.py:516:join_raft_peers
unit-vault-5: 22:09:13 INFO unit.vault/5.juju-log certificates:439: Joining raft cluster address http://10.105.121.5:8200
unit-vault-5: 22:09:43 WARNING unit.vault/5.juju-log certificates:439: Failed to join raft cluster: HTTPConnectionPool(host='127.0.0.1', port=8220): Read timed out. (read timeout=30)
unit-vault-5: 22:09:43 INFO unit.vault/5.juju-log certificates:439: Joining raft cluster address http://10.105.121.6:8200
unit-vault-5: 22:10:13 WARNING unit.vault/5.juju-log certificates:439: Failed to join raft cluster: HTTPConnectionPool(host='127.0.0.1', port=8220): Read timed out. (read timeout=30)
unit-vault-5: 22:10:13 INFO unit.vault/5.juju-log certificates:439: Invoking reactive handler: reactive/vault_handlers.py:690:send_vault_url_and_ca
unit-vault-5: 22:10:13 WARNING unit.vault/5.juju-log certificates:439: Use of remote_binding in publish_url is deprecated. See LP Bug #1895185
unit-vault-5: 22:10:13 INFO unit.vault/5.juju-log certificates:439: Invoking reactive handler: reactive/vault_handlers.py:736:prime_assess_status
unit-vault-5: 22:10:13 INFO unit.vault/5.juju-log certificates:439: Invoking reactive handler: hooks/relations/tls-certificates/provides.py:45:joined:certificates
unit-vault-5: 22:09:12 INFO unit.vault/5.juju-log certificates:439: Reactive main running for hook certificates-relation-joined
unit-vault-5: 22:09:12 ERROR unit.vault/5.juju-log certificates:439: Unable to find implementation for relation: peers of vault-ha
unit-vault-5: 22:09:13 INFO unit.vault/5.juju-log certificates:439: Initializing Leadership Layer (is follower)
unit-vault-5: 22:09:13 INFO unit.vault/5.juju-log certificates:439: Initializing Snap Layer
unit-vault-5: 22:09:13 INFO unit.vault/5.juju-log certificates:439: Invoking reactive handler: reactive/vault_handlers.py:294:configure_vault_raft
unit-vault-5: 22:09:13 INFO unit.vault/5.juju-log certificates:439: Making dir /var/snap/vault/common/data/ root:root 700
unit-vault-5: 22:09:13 INFO unit.vault/5.juju-log certificates:439: Invoking reactive handler: reactive/vault_handlers.py:353:mysql_setup
unit-vault-5: 22:09:13 INFO unit.vault/5.juju-log certificates:439: Invoking reactive handler: reactive/vault_handlers.py:384:database_not_ready
...

syslog:
...
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]: 2023-04-24T22:18:54.585Z [ERROR] core: failed to retry join raft cluster: retry=2s
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]: 2023-04-24T22:18:54.585Z [WARN] core: join attempt failed:
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]: error=
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]: | error during raft bootstrap init call: Error making API request.
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]: |
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]: | URL: PUT http://10.105.121.6:8200/v1/sys/storage/raft/bootstrap/challenge
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]: | Code: 404. Errors:
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]: |
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]: | * unsupported path
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]:
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]: 2023-04-24T22:18:54.585Z [ERROR] core: failed to retry join raft cluster: retry=2s
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]: 2023-04-24T22:18:54.628Z [INFO] core: security barrier not initialized
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]: 2023-04-24T22:18:54.628Z [INFO] core: attempting to join possible raft leader node: leader_addr=http://10.105.121.6:8200
Apr 24 22:18:54 juju-b9f7b8-2-lxd-49 vault[6220]: 2023-04-24T22:18:54.628Z [WARN] core: join attempt failed: error="error during raft bootstrap init call: Put \"http://10.105.121.6:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.105.121.6:8200: socket: too many open files"
...