sunbeam cluster join turns into Error with 'Bootstrapping': Snap: 'openstack-hypervisor'; command ['snap', 'set', ' openstack-hypervisor', '-t', 'rabbitmq.url=null'] failed with output = ''

[Nobuto Murata]

$ snap list openstack
Name Version Rev Tracking Publisher Notes
openstack 2024.1 503 2024.1/edge canonical✓ -

It's eventually getting fine after some time so it shouldn't be critical, but it errors out during the join process. It looks like the charm tries to set "null" to rabbitmq.url and it tries to run the command even efore the snap to get ready to accept commands.

$ sunbeam cluster join --role control --role compute --role storage --token eyJuYW1lIjoic3VuYmVhbS0yLmxvY2FsZG9tYWluIiwic2VjcmV0IjoiNzFjZDBhN2RjYTRjNzU3ODQ0OWRiZTUzNTJhNjgxOTY0NmEzY2E2NDVlZTI1YjRmOWIwZWQwY2UxNzZlYWEyOCIsImZpbmdlcnByaW50IjoiMjIyNTQyYjU3YTVmMGJhN2RmZTMyYjA1NDBjZWQyYzcwNjkzMDc3ZWU1NTg2YmFjZWM1MzI1Mjg4YjA4NGM4OSIsImpvaW5fYWRkcmVzc2VzIjpbIjEwLjAuMTIzLjI0MTo3MDAwIl19

$ juju status -m controller

...

openstack-hypervisor/1 blocked executing 1 10.0.123.234 (workload) Error in charm (see logs): Snap: 'openstack-hypervisor'; command ['snap', 'set', 'openstack-hypervisor', '...

unit-openstack-hypervisor-1: 08:49:25 INFO unit.openstack-hypervisor/1.juju-log Relations {'amqp'} incomplete
unit-openstack-hypervisor-1: 08:49:25 INFO unit.openstack-hypervisor/1.juju-log In stop_services for relation amqp
unit-openstack-hypervisor-1: 08:49:25 DEBUG unit.openstack-hypervisor/1.juju-log Resetting rabbitmq url
unit-openstack-hypervisor-1: 08:49:27 WARNING unit.openstack-hypervisor/1.config-changed error: snap "openstack-hypervisor" has no "rabbitmq" configuration option
unit-openstack-hypervisor-1: 08:49:27 DEBUG unit.openstack-hypervisor/1.juju-log Applying new snap settings {'rabbitmq.url': None}
unit-openstack-hypervisor-1: 08:49:27 WARNING unit.openstack-hypervisor/1.config-changed error: snap "openstack-hypervisor" not found
unit-openstack-hypervisor-1: 08:49:27 ERROR unit.openstack-hypervisor/1.juju-log Exception raised in section 'Bootstrapping': Snap: 'openstack-hypervisor'; command ['snap', 'set', '
openstack-hypervisor', '-t', 'rabbitmq.url=null'] failed with output = ''
unit-openstack-hypervisor-1: 08:49:27 ERROR unit.openstack-hypervisor/1.juju-log Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-openstack-hypervisor-1/charm/lib/charms/operator_libs_linux/v2/snap.py", line 278, in _snap
    return subprocess.check_output(args, universal_newlines=True)
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['snap', 'set', 'openstack-hypervisor', '-t', 'rabbitmq.url=null']' returned non-zero exit status 1.

$ sudo snap set openstack-hypervisor -t rabbitmq.url=null
error: snap "openstack-hypervisor" has "install-snap" change in progress

[Nobuto Murata (nobuto)]

$ juju show-status-log openstack-hypervisor/1 --days 1

...

16 May 2024 08:49:27Z juju-unit executing running start hook
16 May 2024 08:49:48Z juju-unit executing running identity-credentials-relation-changed hook
16 May 2024 08:50:16Z juju-unit executing running receive-ca-cert-relation-joined hook for cert-distributor/0
16 May 2024 08:50:38Z juju-unit executing running receive-ca-cert-relation-changed hook for cert-distributor/0
16 May 2024 08:51:06Z juju-unit executing running nova-service-relation-changed hook
16 May 2024 08:51:34Z juju-unit executing running ovsdb-cms-relation-joined hook for ovn-relay/0
16 May 2024 08:51:55Z juju-unit executing running nova-service-relation-joined hook for nova/0
16 May 2024 08:52:15Z juju-unit executing running amqp-relation-changed hook
16 May 2024 08:52:43Z juju-unit executing running amqp-relation-joined hook for rabbitmq/0
16 May 2024 08:53:04Z juju-unit executing running amqp-relation-changed hook for rabbitmq/0
16 May 2024 08:55:08Z workload blocked (workload) Error in charm (see logs): Snap: 'openstack-hypervisor'; command ['snap', 'set', 'openstack-hypervisor', '-t', 'rabbitmq.url=null'] failed with output = ''
16 May 2024 08:55:09Z juju-unit executing running nova-service-relation-changed hook for nova/0
16 May 2024 08:56:40Z juju-unit executing running ovsdb-cms-relation-changed hook for ovn-relay/0
16 May 2024 08:58:08Z juju-unit executing running ceph-access-relation-changed hook
16 May 2024 08:59:49Z juju-unit executing running certificates-relation-changed hook
16 May 2024 09:01:19Z juju-unit executing running ceph-access-relation-joined hook for cinder-ceph/0
16 May 2024 09:01:40Z juju-unit executing running ceph-access-relation-changed hook for cinder-ceph/0
16 May 2024 09:03:21Z juju-unit executing running identity-credentials-relation-joined hook for keystone/0
16 May 2024 09:03:29Z workload active

[Nobuto Murata (nobuto)]

Isn't this a charm-openstack-hypervisor issue? Setting it back to sunbeam-charms.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/sunbeam-charms/+/920712
Committed: https://opendev.org/openstack/sunbeam-charms/commit/d68e9c757cc8db2b70c22c030e5d21caff1102d1
Submitter: "Zuul (22348)"
Branch: main

commit d68e9c757cc8db2b70c22c030e5d21caff1102d1
Author: Guillaume Boutry <email address hidden>
Date: Wed May 29 09:41:38 2024 +0200

    [hypervisor] prevent unsetting unconfigured values

    Unsetting unconfigured values result in a snap error and will delay the
    hypervisor configuration. When an unset value is detected, only set it
    when the new value is not null.

    Closes-Bug: #2065864
    Change-Id: Id07f70276dab87326d2f2f024a938cc4b35b8cbd