placement not loading CA if vault is deployed in HA configuration

[Alexandru Petcu]

When deploying placement charm in a bundle with Vault in HA configuration (with hacluster), placement charm is getting stuck on running hooks with the following error:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 808, in render_configs
    _render(os.path.basename(conf))
  File "/var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
    charmhelpers.core.templating.render(
  File "/var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 80, in render
    content = template.render(context)
  File "/var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "templates/openstack_https_frontend.conf", line 1, in top-level template code
    {% if options.endpoints -%}
  File "/var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 485, in getattr
    return getattr(obj, attribute)
  File "/var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-packages/charms_openstack/adapters.py", line 1059, in endpoints
    int_port = ch_cluster.determine_api_port(
  File "/var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 265, in determine_api_port
    if https():
  File "/var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 228, in https
    cert_utils.get_requests_for_local_unit("certificates")
  File "/var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 424, in get_requests_for_local_unit
    'ca': data['ca'],
KeyError: 'ca'

This happens when the vault leader is not the first vault charm in the related_units(rid) list, because only the leader charm contains the CA and certificates information.

$juju status placement --relations
Model Controller Cloud/Region Version SLA Timestamp
openstack maas-controller maas/default 2.9.43 unsupported 16:32:29+03:00

App Version Status Scale Charm Channel Rev Exposed Message
placement 9.0.0 error 1 placement 2023.1/stable 87 no hook failed: "shared-db-relation-changed"
placement-mysql-router 8.0.33 active 1 mysql-router 8.0/stable 90 no Unit is ready

Unit Workload Agent Machine Public address Ports Message

placement/9* error idle 3/lxd/20 172.30.193.62 8778/tcp hook failed: "shared-db-relation-changed"
  placement-mysql-router/9* active idle 172.30.193.62 Unit is ready

Machine State Address Inst id Series AZ Message
3 started 172.30.191.11 cloud1 jammy default Deployed
3/lxd/20 started 172.30.193.62 juju-b1ca20-3-lxd-20 jammy default Container started

Relation provider Requirer Interface Type Message
keystone:identity-service placement:identity-service keystone regular
mysql-innodb-cluster:db-router placement-mysql-router:db-router mysql-router regular
placement-mysql-router:shared-db placement:shared-db mysql-shared subordinate
placement:cluster placement:cluster openstack-ha peer
placement:placement nova-cloud-controller:placement placement regular
vault:certificates placement:certificates tls-certificates regular

$ juju status vault --relations
Model Controller Cloud/Region Version SLA Timestamp
openstack maas-controller maas/default 2.9.43 unsupported 16:35:17+03:00

App Version Status Scale Charm Channel Rev Exposed Message
vault 1.7.9 active 3 vault 1.7/stable 107 no Unit is ready (active: true, mlock: disabled)
vault-hacluster active 3 hacluster 2.4/stable 120 no Unit is ready and clustered
vault-mysql-router 8.0.33 active 3 mysql-router 8.0/stable 90 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
vault/0 active idle 3/lxd/14 172.30.193.40 8200/tcp Unit is ready (active: true, mlock: disabled)
  vault-hacluster/1 active idle 172.30.193.40 Unit is ready and clustered
  vault-mysql-router/1 active idle 172.30.193.40 Unit is ready
vault/1 active idle 4/lxd/13 172.30.193.48 8200/tcp Unit is ready (active: false, mlock: disabled)
  vault-hacluster/2 active idle 172.30.193.48 Unit is ready and clustered
  vault-mysql-router/2 active idle 172.30.193.48 Unit is ready
vault/2* active idle 5/lxd/13 172.30.193.2 8200/tcp Unit is ready (active: false, mlock: disabled)
  vault-hacluster/0* active idle 172.30.193.2 Unit is ready and clustered
  vault-mysql-router/0* active idle 172.30.193.2 Unit is ready

Machine State Address Inst id Series AZ Message
3 started 172.30.191.11 cloud1 jammy default Deployed
3/lxd/14 started 172.30.193.40 juju-b1ca20-3-lxd-14 jammy default Container started
4 started 172.30.191.12 cloud2 jammy default Deployed
4/lxd/13 started 172.30.193.48 juju-b1ca20-4-lxd-13 jammy default Container started
5 started 172.30.191.13 cloud3 jammy default Deployed
5/lxd/13 started 172.30.193.2 juju-b1ca20-5-lxd-13 jammy default Container started

Relation provider Requirer Interface Type Message
etcd:db vault:etcd etcd regular
mysql-innodb-cluster:db-router vault-mysql-router:db-router mysql-router regular
vault-hacluster:ha vault:ha hacluster subordinate
vault-hacluster:hanode vault-hacluster:hanode hacluster peer
vault-mysql-router:shared-db vault:shared-db mysql-shared subordinate
vault:certificates ceph-radosgw:certificates tls-certificates regular
vault:certificates cinder:certificates tls-certificates regular
vault:certificates glance:certificates tls-certificates regular
vault:certificates keystone:certificates tls-certificates regular
vault:certificates mysql-innodb-cluster:certificates tls-certificates regular
vault:certificates neutron-api-plugin-ovn:certificates tls-certificates regular
vault:certificates neutron-api:certificates tls-certificates regular
vault:certificates nova-cloud-controller:certificates tls-certificates regular
vault:certificates openstack-dashboard:certificates tls-certificates regular
vault:certificates ovn-central:certificates tls-certificates regular
vault:certificates placement:certificates tls-certificates regular
vault:cluster vault:cluster vault-ha peer

$juju run -u placement/9 -- relation-get -r 126 - vault/0
egress-subnets: 172.30.193.40/32
ingress-address: 172.30.193.40
private-address: 172.30.193.40

$juju run -u placement/9 -- relation-get -r 126 - vault/1
egress-subnets: 172.30.193.48/32
ingress-address: 172.30.193.48
private-address: 172.30.193.48

(this is a test environment with vault generated certificates, so there is no sensitive information)
$ juju run -u placement/9 -- relation-get -r 126 - vault/2
ca: |-
  -----BEGIN CERTIFICATE-----
  MIIDazCCAlOgAwIBAgIUGgHpCm1sEiv7QLXR8i15E02G7a4wDQYJKoZIhvcNAQEL
  BQAwPTE7MDkGA1UEAxMyVmF1bHQgUm9vdCBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkg
  KGNoYXJtLXBraS1sb2NhbCkwHhcNMjMwNzIwMTIwNzA4WhcNMzMwNzE3MTEwNzM3
  WjA9MTswOQYDVQQDEzJWYXVsdCBSb290IENlcnRpZmljYXRlIEF1dGhvcml0eSAo
  Y2hhcm0tcGtpLWxvY2FsKTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
  AL014CuJwXTrDNr5efd6mhokm07mjHnvV074cF07EHwugjGar0zju6u5ccG1qw/o
  iRu/3N1cBmbfJ83LfJnh8XPenOA/2BucSMOwF+mx4EdF8wx1CMtiHXTMtHzetAJE
  e/mFO9vXoc/WV/pLrYm8cy6A41xZ6btQaRHraI+YPUN+peb9tfgfTCL31Zuu8gYI
  VDseoxd74xG20vTD+YUwWxmFfbZt5YiVMHRh0sjsoVGFaLo1pGGjzTzGoJFxh7Ef
  NskKl2eFRbD2EQh0jrVpTCwRGnL1okg3lNCx2tElXFPdzDWuPZFbSgbVioaOgpK1
  /xSkDpGuTFRdcyzemppLomECAwEAAaNjMGEwDgYDVR0PAQH/BAQDAgEGMA8GA1Ud
  EwEB/wQFMAMBAf8wHQYDVR0OBBYEFD4c0UVCNNNqatMDAFuv7g+xmsY8MB8GA1Ud
  IwQYMBaAFD4c0UVCNNNqatMDAFuv7g+xmsY8MA0GCSqGSIb3DQEBCwUAA4IBAQBs
  bAMpuufkAE/kabrRx+D96hxxEZbaxrLS+Ji2uI27JtF2qlqWYRFH/E/nHP0CuW7w
  OOFkAilw/keJ5tvMrszzzUPCTMceV4fT2mwk7S1G6ySt5ctjTyB/Usv9PhgaI+Rn
  8tXci8j0DH9T05NbAxveIeUjHINxpcnJ301p/mn9UF3DBrPd0B1Ex2kGwR+mjCwa
  39mvnB7vTkOVp+im2e1zlk9inPN/SsvrnsgZ02YMCn4AJFVSvxN+pjXgxaedAnbJ
  ILsmpPBtd/fLmc1mc40Tm0RXhlhEaaCH18aGmMtBMi26+sUESeqLh7cpzRr4j1YG
  Vw0PJJey3Z9Ad58fOY69
  -----END CERTIFICATE-----
client.cert: |-
  -----BEGIN CERTIFICATE-----
  MIID+jCCAuKgAwIBAgIUPyvgxUMQO3XN1N5XcuhtZRLHh1owDQYJKoZIhvcNAQEL
  BQAwPTE7MDkGA1UEAxMyVmF1bHQgUm9vdCBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkg
  KGNoYXJtLXBraS1sb2NhbCkwHhcNMjMwNzIxMTE0MjI0WhcNMjQwNzIwMTA0MjUz
  WjAYMRYwFAYDVQQDEw1nbG9iYWwtY2xpZW50MIIBIjANBgkqhkiG9w0BAQEFAAOC
  AQ8AMIIBCgKCAQEAzoJWz5otQFeCTCipDB7B6cGbipR3Q3lKRyWMKAR3o/DLdtnx
  /kN86++dERwWkJ7Ap0KbBiKRxZYX8z6IbRV3VKONiGcJ/qh42OfGWLBRLDPbQjGF
  sYsya7W2mTovabTe9QNu2bVy/yNlK185xUda6rRFjjdIPEx2Wf4KBKcsLQ9iD7Sj
  nCta9RzPJ0eurVAkkAXtwZbt4Yx/g2evuZPG2hzLmVrU1kqhRHlqiPDl05H6rdBl
  IrXBRsRA3MZ1hDFGCjrN4FFLYhjhvv5rIXBQP9O6w6lNvnr4lfFho1eLBYoJpq/F
  1+bjsMIdgC90iaGqNqDgktt9P1lDxbw7TmoXsQIDAQABo4IBFTCCAREwDgYDVR0P
  AQH/BAQDAgOoMBMGA1UdJQQMMAoGCCsGAQUFBwMCMB0GA1UdDgQWBBTx2phoE2+t
  Qnq9MKDJU+3TxRs+AzAfBgNVHSMEGDAWgBQ+HNFFQjTTamrTAwBbr+4PsZrGPDBM
  BggrBgEFBQcBAQRAMD4wPAYIKwYBBQUHMAKGMGh0dHA6Ly8xNzIuMzAuMTkzLjEw
  ODo4MjAwL3YxL2NoYXJtLXBraS1sb2NhbC9jYTAYBgNVHREEETAPgg1nbG9iYWwt
  Y2xpZW50MEIGA1UdHwQ7MDkwN6A1oDOGMWh0dHA6Ly8xNzIuMzAuMTkzLjEwODo4
  MjAwL3YxL2NoYXJtLXBraS1sb2NhbC9jcmwwDQYJKoZIhvcNAQELBQADggEBAJZw
  0cPh/kENIJ6pzKmiYzPjT2jlRX5+xtvL622xZF4aJCKyNefH0f7XVAdqg10x/Vts
  CE259VarF7Mwo9oTFIlPJkmnL91mAPL6oZj1r7/HRRI4ujBWBBMzp2SzM6xGGpNS
  itKNiyaWJm71YaO0x16yfC4aFfnqLJs9w3TCgS3zY3hiIDAYbu1DPJaeJCpGXMRi
  rYrEgtzi5qwBPKlI/f06iC6WU9aYTROClQWpMktKrq022/S/dgMkxNJXbIBeMjFL
  MCEjL2iRzmmy/MxGapz5lCu0sM2L7Ufbi0LoKhR5yk3yTFrTEecnpExLhB0R+pdP
  9s7i+N/I6AuumQpP5sg=
  -----END CERTIFICATE-----
client.key: |-
  -----BEGIN RSA PRIVATE KEY-----
  MIIEpAIBAAKCAQEAzoJWz5otQFeCTCipDB7B6cGbipR3Q3lKRyWMKAR3o/DLdtnx
  /kN86++dERwWkJ7Ap0KbBiKRxZYX8z6IbRV3VKONiGcJ/qh42OfGWLBRLDPbQjGF
  sYsya7W2mTovabTe9QNu2bVy/yNlK185xUda6rRFjjdIPEx2Wf4KBKcsLQ9iD7Sj
  nCta9RzPJ0eurVAkkAXtwZbt4Yx/g2evuZPG2hzLmVrU1kqhRHlqiPDl05H6rdBl
  IrXBRsRA3MZ1hDFGCjrN4FFLYhjhvv5rIXBQP9O6w6lNvnr4lfFho1eLBYoJpq/F
  1+bjsMIdgC90iaGqNqDgktt9P1lDxbw7TmoXsQIDAQABAoIBAQCNhoHcEPuGJvLj
  ItKHveSDQO5KEOmTUmJ3U+Iz4qvjrLikIUVZTgMk5rgr66YDQukhN+JqV/KzCOQt
  DeQNbjKzzMrIemU9I1R+wokKavuWYYuYV7waKx1I9+XGKTwpI43ROKMVjObct/Mz
  wDUlYHuMcjNg/b4BpR1PG2zeN3ts/OvSTzZruqgKwfIMPKrSGoiApGjY0uGY84UV
  FL1Z2ymSBWP09LE5/e3Yf60gPElCJrgdhzHsHvx5O0RGKQhqa1yxB1ohrguRBXOO
  Gpl8U5A2xPrPcJTiTVDwvFr8LsOeg4K7V9V18nZoiul+vHoYsSXhR2TD4FuG15Jx
  xLq4/rrhAoGBANT/6Ip83yPOOqJxemCSTQZTLHmMw7HgdLzhecsFz77Cs4aWwIcn
  Aw7OHG7jY47ThJrcs8V/Pn+Q9of8PEHscEAf7irzL1u9iUzHscDe+riBD0E+d7jH
  jR6/ncDdRkqBKCsjet6WwbL/L5IdZgv6cfeQvqd3ocyGkIH25DtvFaN9AoGBAPgy
  /kNxhyzfcUe54QJIWhfkkOYL+HxbExkA7D/WnH1i3u+dMy6VMNeh6YqwXLwoTeIF
  cJK3H/ptbeOMh22buOlkK/4shkeEFs/HZcrEEP/7HCawTIRoSQR+fFN4XpCjKoRn
  z25lxcMvB0xwgxGBbm0nYB6eMEQrF7+rKvZ1OdNFAoGAO+iSfkkq9cYuhbwHIJxl
  n67LJpDqBeDM1MdSW0ESMmRO3D8QfbMCUCq2opXATc6+UFRdF+J3MKgUPyE5zYzz
  8YVJlrj5jAMeWAMewtG0kSpzoPUx9T3jOk/IzhMujwwOwTtkdrDOYEsYFS/BoQLb
  BTen+DBUyNLcWEGZ5gGTg30CgYBXFP0lCKYW1mbWdZ8+v8fsbiNws26Boyq2MR6y
  30eUrqXtiEg25d13KrXYFTzUP4fj72ip8Vpk8RQHLH0vtd1Q9faBfXm3id6/FATg
  +Z3FUoN0B5MxOa20x/QzDR6QvM7kGdArucWDT/LNnQQ/lhv6bEBLPdFNeZkhhJrU
  UhjxyQKBgQCBUFgP1eWGL5gOHBApxRkZdHVr2oziXxkSdiI8Qol+pGWq04G4d5Cl
  QUwfkIGegLDRl7Wg+s9NFFwz/nEzevba8vVPCIASKNCrkNFvRGA6vfbrVGxq3gKH
  uTUFwD/XPE03usGa4wjgWS/tIzoUL1QEVc0W/Br3sFfT3fBQ4jxyZg==
  -----END RSA PRIVATE KEY-----
egress-subnets: 172.30.193.2/32
ingress-address: 172.30.193.2
placement_9.server.cert: |-
  -----BEGIN CERTIFICATE-----
  MIIEHjCCAwagAwIBAgIUW6ygfrOriq2GonP4tZfMlyyStYcwDQYJKoZIhvcNAQEL
  BQAwPTE7MDkGA1UEAxMyVmF1bHQgUm9vdCBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkg
  KGNoYXJtLXBraS1sb2NhbCkwHhcNMjMwNzIxMTE0MjI1WhcNMjQwNzIwMTA0MjU1
  WjAfMR0wGwYDVQQDExRqdWp1LWIxY2EyMC0zLWx4ZC0yMDCCASIwDQYJKoZIhvcN
  AQEBBQADggEPADCCAQoCggEBAMBH00iy+95YY4HFqq4RtLpki3Oc33cwePhD2bDN
  Dagm31xHUPL/HhXu7OrJgPCFHQNqEJ1a/sXn5125sucVKcmBQG02iFb3EDD2zm9l
  jKAYxMATZB1Fv97IwIQm+MixzHV66btroe4AdX/9vzTRRr12qb0rU7yQJNhfp6jh
  ytSj4NoD1WZ0AGBziacTa5aSgaarGFIHth78ujHKN/dqa8O5aEBfH0S+HoDuzRNJ
  6N1oThm1APrHCB9bojpZ7jz20jJpcdmWQW/2aqvQY1ZDgBBvfYjwQW0WH2XQKpe5
  ZO2id3gk+js6fenCVg9NnGnGOR9nfCrnkNMQtVpEdpapry0CAwEAAaOCATIwggEu
  MA4GA1UdDwEB/wQEAwIDqDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIw
  HQYDVR0OBBYEFJMG+P2DNrOI++LwWNaU2xI1LR3HMB8GA1UdIwQYMBaAFD4c0UVC
  NNNqatMDAFuv7g+xmsY8MEwGCCsGAQUFBwEBBEAwPjA8BggrBgEFBQcwAoYwaHR0
  cDovLzE3Mi4zMC4xOTMuMTA4OjgyMDAvdjEvY2hhcm0tcGtpLWxvY2FsL2NhMCsG
  A1UdEQQkMCKCFGp1anUtYjFjYTIwLTMtbHhkLTIwhwSsHsE+hwSsHsIvMEIGA1Ud
  HwQ7MDkwN6A1oDOGMWh0dHA6Ly8xNzIuMzAuMTkzLjEwODo4MjAwL3YxL2NoYXJt
  LXBraS1sb2NhbC9jcmwwDQYJKoZIhvcNAQELBQADggEBAAXgqRGIvdpr8e9cycHA
  4FBBOjNn6/ugKFXaINNYpvs/WOqpqGyNKCrjGKVG0sn+w9inyefMSkhVex9f31aZ
  wNIHimEn/zSUDscIc16aAI+k17Z4nTGFibHMIl7MLEcZePQS+FnnQtzLu7C8afQv
  aYUxkNMbrjmGd/UmPBI1D7PtLajlg6e4AQxkHi8d0X8zvYTLgOCJ0pMzMdpf+Qse
  AxtiUNiW46ccuZ99bsKzlfeqcaR0v46l1zn87CiBlgbsf2XF4OE1Pc+kN0WT/wNl
  4powNzCxhZXFTxS6TnvY44XMsMC6EYMObELMMGHJ8T5dF6MdBMjSsYzxBjiUCwJx
  SKc=
  -----END CERTIFICATE-----
placement_9.server.key: |-
  -----BEGIN RSA PRIVATE KEY-----
  MIIEpAIBAAKCAQEAwEfTSLL73lhjgcWqrhG0umSLc5zfdzB4+EPZsM0NqCbfXEdQ
  8v8eFe7s6smA8IUdA2oQnVr+xefnXbmy5xUpyYFAbTaIVvcQMPbOb2WMoBjEwBNk
  HUW/3sjAhCb4yLHMdXrpu2uh7gB1f/2/NNFGvXapvStTvJAk2F+nqOHK1KPg2gPV
  ZnQAYHOJpxNrlpKBpqsYUge2Hvy6Mco392prw7loQF8fRL4egO7NE0no3WhOGbUA
  +scIH1uiOlnuPPbSMmlx2ZZBb/Zqq9BjVkOAEG99iPBBbRYfZdAql7lk7aJ3eCT6
  Ozp96cJWD02cacY5H2d8KueQ0xC1WkR2lqmvLQIDAQABAoIBAQCA86qPJvojs8OI
  VLiS5Oz/1E5xytNTa/61AUwNdneiPe+uQ2XWkaHrs8DP9TO0GpHKqGdcIJqcv1o7
  UT7aGydbLIO4usnVCPyi2eK4crJM7E+HQXny58Q19QynlDtk3VjxsOzv9M80frpO
  jC6fjXqTtfFAHL6bx0/ujLg0WrlcwSF1hHo1YpWg+CZ6aIRxrg6+9KHBiDxdQwjZ
  RShsKtdISU+hNSC5Iwami+ZvT/iWxuXpeXAYFyTw2plAknSLt5Dq1pHXPwHG1hPw
  js8m/P/05jyVG49qzsmHNtARxzS1SxUKzyF18T826vCei1NzgVg5KHPVCqakXU+C
  sFoPkvFBAoGBAOJKeSZ5leWCV/q7oXiWGboRNGYIrOUcGxnArF0E28+xiR+unpEx
  e7FZQCbSz/f7Jkgy9KmUA0XObAl5GBpSSKxlJKaKHCmP4cJzRM4jwd/y6YhWiVgR
  FEZQsDMXNHYQCZVSQzQ4T+9/xFCwynk459eKRbZ8jIPNvJMrHp8Js4PHAoGBANmG
  R/WB/pCkbzOrVaAYIM2opENx1dfIREg1w7pqYjraXOBpdxulFYJBBAgJ696UKl8V
  oYg5teQBWiZvjWCOEcpF0Pmjz69HeowJvx63PCgSfY/UmlPXZ5bJowvcdT0q8sEH
  waTOJ0mu/SNqRzP47NnVPpYTAzIGJPSc7fIEi41rAoGAPhUof8HDuh+0e3mWpzqm
  /EFUGtoudPztAne10h9PHwAaGrMRlcv5n4JY803K+MHSkuRH825EJ5amVUL/tdj1
  TzbAg2BIsa5jxbFSIyv0OLiwEVOMgrjM0y236YxoPCvSNUFIZr88TvzCo1Cde/x8
  vsmUrqgk9NQw9D9UO1WyBmkCgYEApDx41GskE32z4GU6rzfIfL8Z1zp55QR2WbcO
  Yw+zvw/kFCRNI/hO9xuzuYYUKuff3g7s+XfpV4+Vy3NqYT6jO//eur3uLz9ezaEG
  RmC+8NqVJyW0qBz5P6ZajrgTZO2qgAglMxPcIgwXYMf7nsgRvJ7/NDyCqyBsgqyC
  E7tOyqUCgYAYYF1S1f6PI9xjoi1drUs/35kC+UBdq0JaseoFCBFqwg2/HVpx214e
  PMpxhPXWUjtTu1hecKAfl5gAmGg8qdBNe8ZSWo1065R7+4JPNf18Vjh2QXMIVeiN
  FuPfpfhfqgmI1uFIUzqYs/gGYOakEEYeM/TaySEmKKNUnhqizWQFBA==
  -----END RSA PRIVATE KEY-----
private-address: 172.30.193.2

Checking the python script which is generating the error, we can see that the failure happens because there is no data['ca'] available in the first vault unit.

I have made a small modification in my /var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-packages/charmhelpers/contrib/openstack/cert_utils.py which fixed my deployment. This change is not good for a different configuration, because it will actually bypass loading the CA if it not available on any vault unit.
' if 'ca' in data.keys(): '

Original function:

def get_requests_for_local_unit(relation_name=certificates):
    """Extract any certificates data targeted at this unit down relation_name.

    :param relation_name: str Name of relation to check for data.
    :returns: List of bundles of certificates.
    :rtype: List of dicts
    """
    local_name = local_unit().replace('/', '_')
    raw_certs_key = '{}.processed_requests'.format(local_name)
    relation_name = relation_name or 'certificates'
    bundles = []
    for rid in relation_ids(relation_name):
        sent = relation_get(rid=126, unit=local_unit())
        legacy_keys = ['certificate_name', 'common_name']
        is_legacy_request = set(sent).intersection(legacy_keys)
        for unit in related_units(rid):
            data = relation_get(rid=rid, unit=unit)
                if data.get(raw_certs_key):
                    bundles.append({
                        'ca': data['ca'],
                        'chain': data.get('chain'),
                        'certs': json.loads(data[raw_certs_key])})
                elif is_legacy_request:
                    bundles.append({
                        'ca': data['ca'],
                        'chain': data.get('chain'),
                        'certs': {sent['common_name']:
                                {'cert': data.get(local_name + '.server.cert'),
                                'key': data.get(local_name + '.server.key')}}})

    return bundles

Modified function:

def get_requests_for_local_unit(relation_name=certificates):
    """Extract any certificates data targeted at this unit down relation_name.

    :param relation_name: str Name of relation to check for data.
    :returns: List of bundles of certificates.
    :rtype: List of dicts
    """
    local_name = local_unit().replace('/', '_')
    raw_certs_key = '{}.processed_requests'.format(local_name)
    relation_name = relation_name or 'certificates'
    bundles = []
    for rid in relation_ids(relation_name):
        sent = relation_get(rid=126, unit=local_unit())
        legacy_keys = ['certificate_name', 'common_name']
        is_legacy_request = set(sent).intersection(legacy_keys)
        for unit in related_units(rid):
            data = relation_get(rid=rid, unit=unit)
            if 'ca' in data.keys():
                if data.get(raw_certs_key):
                    bundles.append({
                        'ca': data['ca'],
                        'chain': data.get('chain'),
                        'certs': json.loads(data[raw_certs_key])})
                elif is_legacy_request:
                    bundles.append({
                        'ca': data['ca'],
                        'chain': data.get('chain'),
                        'certs': {sent['common_name']:
                                {'cert': data.get(local_name + '.server.cert'),
                                'key': data.get(local_name + '.server.key')}}})

    return bundles

This "fix" would probably have been easier to do by modifying "ca: data['ca']" to "ca: data.get('ca')", but I wanted a line which I can easily comment.

Thank you for your support,
Alex

[Alexandru Petcu (alexp32)]

Hi,

An update:

This is probably a Juju issue, as I have encountered the same problem with Magnum and Gnocchi charms as well.

Regards,
Alex

[Felipe Reyes (freyes)]

On Fri, 2023-07-21 at 14:10 +0000, Alexandru Petcu wrote:
>
> [...]
>   File "/var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-
> packages/charmhelpers/contrib/hahelpers/cluster.py", line 265, in determine_api_port
>     if https():
>   File "/var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-
> packages/charmhelpers/contrib/hahelpers/cluster.py", line 228, in https
>     cert_utils.get_requests_for_local_unit("certificates")
>   File "/var/lib/juju/agents/unit-placement-9/.venv/lib/python3.10/site-
> packages/charmhelpers/contrib/openstack/cert_utils.py", line 424, in get_requests_for_local_unit
>     'ca': data['ca'],
> KeyError: 'ca'

We fixed recently bug 2028683 which I believe is the same issue described here.

https://github.com/juju/charm-helpers/commit/c8f47207c1f1652ab2cbecaaad8a83b66f004e5e

The backports haven't landed yet in stable releases yet, but they will.