When trying to bootstrap OpenStack using Sunbeam, there is an error in the charm of neutron, which looks like the following:
* In app view:
neutron waiting 1 neutron-k8s 2023.1/stable 53 10.152.183.36 no installing agent
* In unit view:
neutron/0* blocked idle 10.1.163.29 (workload) Error in charm (see logs): timed out waiting for change 2 (301 seconds)
* Looking at the logs for the error, they look like this:
2023-11-30T09:27:16.903Z [container-agent] 2023-11-30 09:27:16 INFO juju-log identity-service:85: Syncing database...
2023-11-30T09:32:18.000Z [container-agent] 2023-11-30 09:32:18 ERROR juju-log identity-service:85: Exception raised in section 'Bootstrapping': timed out waiting for change 2 (301 seconds)
2023-11-30T09:32:18.008Z [container-agent] 2023-11-30 09:32:18 ERROR juju-log identity-service:85: Traceback (most recent call last):
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/guard.py", line 91, in guard
2023-11-30T09:32:18.008Z [container-agent] yield
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/charm.py", line 265, in configure_charm
2023-11-30T09:32:18.008Z [container-agent] self.configure_unit(event)
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/charm.py", line 479, in configure_unit
2023-11-30T09:32:18.008Z [container-agent] self.run_db_sync()
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/job_ctrl.py", line 74, in wrapped_f
2023-11-30T09:32:18.008Z [container-agent] f(charm, *args, **kwargs)
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/./src/charm.py", line 302, in run_db_sync
2023-11-30T09:32:18.008Z [container-agent] super().run_db_sync()
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/job_ctrl.py", line 74, in wrapped_f
2023-11-30T09:32:18.008Z [container-agent] f(charm, *args, **kwargs)
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/charm.py", line 549, in run_db_sync
2023-11-30T09:32:18.008Z [container-agent] self._retry_db_sync(cmd)
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/tenacity/__init__.py", line 289, in wrapped_f
2023-11-30T09:32:18.008Z [container-agent] return self(f, *args, **kw)
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/tenacity/__init__.py", line 379, in __call__
2023-11-30T09:32:18.008Z [container-agent] do = self.iter(retry_state=retry_state)
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/tenacity/__init__.py", line 314, in iter
2023-11-30T09:32:18.008Z [container-agent] return fut.result()
2023-11-30T09:32:18.008Z [container-agent] File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-11-30T09:32:18.008Z [container-agent] return self.__get_result()
2023-11-30T09:32:18.008Z [container-agent] File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-11-30T09:32:18.008Z [container-agent] raise self._exception
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/tenacity/__init__.py", line 382, in __call__
2023-11-30T09:32:18.008Z [container-agent] result = fn(*args, **kwargs)
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/charm.py", line 529, in _retry_db_sync
2023-11-30T09:32:18.008Z [container-agent] out, warnings = process.wait_output()
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops/pebble.py", line 1354, in wait_output
2023-11-30T09:32:18.008Z [container-agent] exit_code: int = self._wait()
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops/pebble.py", line 1294, in _wait
2023-11-30T09:32:18.008Z [container-agent] change = self._client.wait_change(self._change_id, timeout=timeout)
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops/pebble.py", line 1794, in wait_change
2023-11-30T09:32:18.008Z [container-agent] return self._wait_change_using_wait(change_id, timeout)
2023-11-30T09:32:18.008Z [container-agent] File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops/pebble.py", line 1820, in _wait_change_using_wait
2023-11-30T09:32:18.008Z [container-agent] raise TimeoutError(f'timed out waiting for change {change_id} ({timeout} seconds)')
2023-11-30T09:32:18.008Z [container-agent] ops.pebble.TimeoutError: timed out waiting for change 2 (301 seconds)
2023-11-30T09:32:18.008Z [container-agent]
2023-11-30T09:33:09.704Z [container-agent] 2023-11-30 09:33:09 INFO juju.worker.uniter.operation runhook.go:186 ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
2023-11-30T09:33:14.730Z [container-agent] 2023-11-30 09:33:14 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)
* This has not allowed me to get past the bootstrapping process to start using OpenStack as 22 of the 23 dependancies are up only.
I am running everything with the following commands:
--> sudo snap install openstack
--> sunbeam prepare-node-script | bash -x && newgrp snap_daemon
--> sunbeam cluster bootstrap --accept-defaults
This does not really apear as a neutron problem now, and should be investigated from Sunbeam's perspective until the problem is narrowed down to something wrong with Neutron.