sunbeam cluster resize fails after adding new nodes

[Bartosz Woronicz]

It seems similar issue to : https://bugs.launchpad.net/snap-openstack/+bug/2026596

This is the situation
ubuntu@node01:~$ sunbeam cluster list
┏━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ Node ┃ Status ┃ Control ┃ Compute ┃ Storage ┃
┡━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ node01.maas │ up │ x │ x │ │
│ node02.maas │ up │ x │ x │ x │
│ node03.maas │ up │ x │ x │ x │
│ node04.maas │ up │ x │ x │ x │
└─────────────┴────────┴─────────┴─────────┴─────────┘

I tried to to subbeam cluster resize after adding 3 nodes (2,3,4)
Attached output with terraform failure
...
Error: base required for Charmhub charms (bad request)

  with module.rabbitmq.juju_application.rabbitmq,
  on modules/rabbitmq/main.tf line 28, in resource "juju_application" "rabbitmq":
  28: resource "juju_application" "rabbitmq" {

Error resizing control plane
Traceback (most recent call last):
  File "/snap/openstack/225/lib/python3.10/site-packages/sunbeam/commands/terraform.py", line 197, in apply
    process = subprocess.run(
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/snap/openstack/225/bin/terraform', 'apply', '-auto-approve', '-no-color', '-parallelism=1']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/openstack/225/lib/python3.10/site-packages/sunbeam/commands/openstack.py", line 358, in run
    self.tfhelper.apply()
  File "/snap/openstack/225/lib/python3.10/site-packages/sunbeam/commands/terraform.py", line 211, in apply
    raise TerraformException(str(e))
sunbeam.commands.terraform.TerraformException: Command '['/snap/openstack/225/bin/terraform', 'apply', '-auto-approve', '-no-color', '-parallelism=1']' returned non-zero exit status 1.
Error: Command '['/snap/openstack/225/bin/terraform', 'apply', '-auto-approve', '-no-color', '-parallelism=1']' returned non-zero exit status 1.

I can see that ovn-relay got issues with certificates relation

2023-07-20 13:28:57,388 DEBUG No binding found for public
2023-07-20 13:28:57,401 DEBUG Setting up certificates event handler
2023-07-20 13:28:57,532 DEBUG Cannot add handler for relation identity-credentials, relation not presen
t in charm metadata
2023-07-20 13:28:57,569 DEBUG Emitting Juju event certificates_relation_joined.
2023-07-20 13:28:57,581 ERROR Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ovn-relay-1/charm/./src/charm.py", line 174, in <module>
    main(OVNRelayOperatorCharm)
  File "/var/lib/juju/agents/unit-ovn-relay-1/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-ovn-relay-1/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-ovn-relay-1/charm/venv/ops/framework.py", line 354, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-ovn-relay-1/charm/venv/ops/framework.py", line 830, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-ovn-relay-1/charm/venv/ops/framework.py", line 919, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-ovn-relay-1/charm/venv/ops_sunbeam/relation_handlers.py", line 901, in _
on_certificates_relation_joined
    private_key=private_key.encode(),
AttributeError: 'NoneType' object has no attribute 'encode'

ubuntu@node01:~$ juju status -m openstack ovn-relay/1
Model Controller Cloud/Region Version SLA Timestamp
openstack sunbeam-controller sunbeam-microk8s/localhost 3.2.0 unsupported 13:26:20Z

App Version Status Scale Charm Channel Rev Address Exposed Message
ovn-relay waiting 1/3 ovn-relay-k8s 23.03/stable 34 10.20.21.12 no installing age
nt

Unit Workload Agent Address Ports Message
ovn-relay/1 error idle 10.1.248.205 hook failed: "certificates-relation-joined"

Offer Application Charm Rev Connected Endpoint
     Interface Role
certificate-authority certificate-authority tls-certificates-operator 22 1/1 certificates
     tls-certificates provider
keystone keystone keystone-k8s 102 1/1 identity-credenti
als keystone-credentials provider
ovn-relay ovn-relay ovn-relay-k8s 34 1/1 ovsdb-cms-relay
     ovsdb-cms provider
rabbitmq rabbitmq rabbitmq-k8s 23 1/1 amqp
     rabbitmq provider

[Bartosz Woronicz (mastier1)]

ubuntu@node01:~$ snap info openstack | grep -E '(installed|tracking)'
tracking: 2023.1/candidate
installed: 2023.1 (225) 119MB -

[James Page (james-page)]

bug 2026596 relates to an issue with DNS resolution between ovn-relay and ovn-central.

Your terraform log is interesting - I've not seen this issue (and I'm running with the same snap version you are).