unit-cinder-ceph-0: 09:54:05 INFO juju.cmd running containerAgent [3.2.0 c7107ada8c471aa3ba105e5433e61861227e2ed4 gc go1.20.4] unit-cinder-ceph-0: 09:54:05 INFO juju.cmd.containeragent.unit start "unit" unit-cinder-ceph-0: 09:54:05 INFO juju.worker.upgradesteps upgrade steps for 3.2.0 have already been run. unit-cinder-ceph-0: 09:54:05 INFO juju.worker.probehttpserver starting http server on 127.0.0.1:65301 unit-cinder-ceph-0: 09:54:05 INFO juju.api connection established to "wss://10.42.0.1:17070/model/5d4e49ff-0ec3-4929-8add-c9ec291f8f4c/api" unit-cinder-ceph-0: 09:54:05 INFO juju.worker.apicaller [5d4e49] "unit-cinder-ceph-0" successfully connected to "10.42.0.1:17070" unit-cinder-ceph-0: 09:54:05 INFO juju.api connection established to "wss://10.42.0.1:17070/model/5d4e49ff-0ec3-4929-8add-c9ec291f8f4c/api" unit-cinder-ceph-0: 09:54:05 INFO juju.worker.apicaller [5d4e49] "unit-cinder-ceph-0" successfully connected to "10.42.0.1:17070" unit-cinder-ceph-0: 09:54:05 INFO juju.worker.migrationminion migration phase is now: NONE unit-cinder-ceph-0: 09:54:05 INFO juju.worker.logger logger worker started unit-cinder-ceph-0: 09:54:05 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: "" unit-cinder-ceph-0: 09:54:05 INFO juju.worker.caasupgrader abort check blocked until version event received unit-cinder-ceph-0: 09:54:05 INFO juju.worker.caasupgrader unblocking abort check unit-cinder-ceph-0: 09:54:05 INFO juju.worker.leadership cinder-ceph/0 promoted to leadership of cinder-ceph unit-cinder-ceph-0: 09:54:05 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-cinder-ceph-0 unit-cinder-ceph-0: 09:54:05 INFO juju.worker.uniter unit "cinder-ceph/0" started unit-cinder-ceph-0: 09:54:05 INFO juju.worker.uniter resuming charm install unit-cinder-ceph-0: 09:54:05 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/cinder-ceph-k8s-38 from API server unit-cinder-ceph-0: 09:54:05 INFO juju.downloader downloading from ch:amd64/jammy/cinder-ceph-k8s-38 unit-cinder-ceph-0: 09:54:05 INFO juju.downloader download complete ("ch:amd64/jammy/cinder-ceph-k8s-38") unit-cinder-ceph-0: 09:54:05 INFO juju.downloader download verified ("ch:amd64/jammy/cinder-ceph-k8s-38") unit-cinder-ceph-0: 09:55:30 INFO juju.worker.uniter hooks are retried true unit-cinder-ceph-0: 09:55:30 INFO juju.worker.uniter found queued "install" hook unit-cinder-ceph-0: 09:55:31 INFO unit.cinder-ceph/0.juju-log Running legacy hooks/install. unit-cinder-ceph-0: 09:55:33 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:35 INFO unit.cinder-ceph/0.juju-log peers:47: Peers on_created unit-cinder-ceph-0: 09:55:35 INFO juju.worker.uniter.operation ran "peers-relation-created" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:37 INFO juju.worker.uniter.operation ran "storage-backend-relation-created" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:38 INFO juju.worker.uniter.operation ran "ceph-relation-created" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:40 INFO juju.worker.uniter.operation ran "database-relation-created" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:42 INFO juju.worker.uniter.operation ran "amqp-relation-created" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:42 INFO juju.worker.uniter found queued "leader-elected" hook unit-cinder-ceph-0: 09:55:44 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:45 INFO unit.cinder-ceph/0.juju-log Entering guarded section: 'Bootstrapping' unit-cinder-ceph-0: 09:55:45 INFO unit.cinder-ceph/0.juju-log Relations {'ceph', 'database', 'amqp'} incomplete unit-cinder-ceph-0: 09:55:45 WARNING unit.cinder-ceph/0.juju-log Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready' unit-cinder-ceph-0: 09:55:46 INFO juju.worker.uniter.operation ran "cinder-volume-pebble-ready" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:47 INFO unit.cinder-ceph/0.juju-log Entering guarded section: 'Bootstrapping' unit-cinder-ceph-0: 09:55:47 INFO unit.cinder-ceph/0.juju-log Relations {'ceph', 'amqp', 'database'} incomplete unit-cinder-ceph-0: 09:55:47 WARNING unit.cinder-ceph/0.juju-log Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready' unit-cinder-ceph-0: 09:55:47 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:47 INFO juju.worker.uniter found queued "start" hook unit-cinder-ceph-0: 09:55:48 INFO unit.cinder-ceph/0.juju-log Running legacy hooks/start. unit-cinder-ceph-0: 09:55:49 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:51 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:52 INFO juju.worker.uniter.operation ran "storage-backend-relation-joined" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:54 INFO juju.worker.uniter.operation ran "amqp-relation-joined" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:56 INFO juju.worker.uniter.operation ran "database-relation-joined" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:57 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:55:59 INFO juju.worker.uniter.operation ran "storage-backend-relation-changed" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:56:00 INFO unit.cinder-ceph/0.juju-log amqp:94: Entering guarded section: 'Bootstrapping' unit-cinder-ceph-0: 09:56:00 INFO unit.cinder-ceph/0.juju-log amqp:94: Relations {'database', 'ceph'} incomplete unit-cinder-ceph-0: 09:56:00 WARNING unit.cinder-ceph/0.juju-log amqp:94: Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready' unit-cinder-ceph-0: 09:56:00 INFO juju.worker.uniter.operation ran "amqp-relation-changed" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:56:02 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:56:03 INFO unit.cinder-ceph/0.juju-log amqp:94: Entering guarded section: 'Bootstrapping' unit-cinder-ceph-0: 09:56:03 INFO unit.cinder-ceph/0.juju-log amqp:94: Relations {'database', 'ceph'} incomplete unit-cinder-ceph-0: 09:56:03 WARNING unit.cinder-ceph/0.juju-log amqp:94: Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready' unit-cinder-ceph-0: 09:56:04 INFO juju.worker.uniter.operation ran "amqp-relation-changed" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:56:05 INFO unit.cinder-ceph/0.juju-log ceph:92: emiting broker_available unit-cinder-ceph-0: 09:56:05 INFO unit.cinder-ceph/0.juju-log ceph:92: add_op_create_replicated_pool: [] unit-cinder-ceph-0: 09:56:05 INFO unit.cinder-ceph/0.juju-log ceph:92: Adding add_op_create_replicated_pool request unit-cinder-ceph-0: 09:56:05 INFO unit.cinder-ceph/0.juju-log ceph:92: Storing request unit-cinder-ceph-0: 09:56:05 INFO unit.cinder-ceph/0.juju-log ceph:92: Calling send_request_if_needed unit-cinder-ceph-0: 09:56:05 INFO juju.worker.uniter.operation ran "ceph-relation-joined" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:56:07 INFO unit.cinder-ceph/0.juju-log ceph:92: ceph client on_changed unit-cinder-ceph-0: 09:56:07 INFO juju.worker.uniter.operation ran "ceph-relation-changed" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:56:16 INFO unit.cinder-ceph/0.juju-log ceph:92: ceph client on_changed unit-cinder-ceph-0: 09:56:16 INFO unit.cinder-ceph/0.juju-log ceph:92: get_existing_request unit-cinder-ceph-0: 09:56:16 INFO unit.cinder-ceph/0.juju-log ceph:92: Json request: {"api-version": 1, "ops": [{"op": "create-pool", "name": "cinder-ceph", "replicas": 1, "pg_num": null, "crush-profile": null, "app-name": "rbd", "compression-algorithm": null, "compression-mode": null, "compression-required-ratio": null, "compression-min-blob-size": null, "compression-min-blob-size-hdd": null, "compression-min-blob-size-ssd": null, "compression-max-blob-size": null, "compression-max-blob-size-hdd": null, "compression-max-blob-size-ssd": null, "group": null, "max-bytes": null, "max-objects": null, "group-namespace": null, "rbd-mirroring-mode": "pool", "weight": 40}], "request-id": "9dd261d4c90d7ba713705ae1b696aecf90f4c7af"} unit-cinder-ceph-0: 09:56:16 INFO unit.cinder-ceph/0.juju-log ceph:92: emiting pools available unit-cinder-ceph-0: 09:56:16 INFO unit.cinder-ceph/0.juju-log ceph:92: Entering guarded section: 'Bootstrapping' unit-cinder-ceph-0: 09:56:16 INFO unit.cinder-ceph/0.juju-log ceph:92: Relations {'database'} incomplete unit-cinder-ceph-0: 09:56:16 WARNING unit.cinder-ceph/0.juju-log ceph:92: Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready' unit-cinder-ceph-0: 09:56:17 INFO juju.worker.uniter.operation ran "ceph-relation-changed" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:58:08 INFO unit.cinder-ceph/0.juju-log database:60: database created at 2023-10-25 07:58:08.590561 unit-cinder-ceph-0: 09:58:08 INFO unit.cinder-ceph/0.juju-log database:60: Received data: {'data': '{"database": "cinder"}', 'database': 'cinder', 'endpoints': 'cinder-ceph-mysql-router.openstack.svc.cluster.local:6446', 'password': 'REDACTED', 'read-only-endpoints': 'cinder-ceph-mysql-router.openstack.svc.cluster.local:6447', 'username': 'relation-85-60'} unit-cinder-ceph-0: 09:58:08 INFO unit.cinder-ceph/0.juju-log database:60: Entering guarded section: 'Bootstrapping' unit-cinder-ceph-0: 09:58:10 ERROR unit.cinder-ceph/0.juju-log database:60: Exception raised in section 'Bootstrapping': cannot perform the following tasks: - Start service "cinder-volume" (cannot start service: exited quickly with code 1) ----- Logs from task 0 ----- 2023-10-25T07:58:10Z INFO Most recent service output: (...) 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 548, in query 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 775, in _read_query_result 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume result.read() 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1156, in read 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume first_packet = self.connection._read_packet() 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 725, in _read_packet 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume packet.raise_for_error() 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 221, in raise_for_error 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume err.raise_mysql_exception(self._data) 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume raise errorclass(errno, errval) 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume [SQL: SELECT services.created_at AS services_created_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.uuid AS services_uuid, services.cluster_name AS services_cluster_name, services.host AS services_host, services.`binary` AS services_binary, services.updated_at AS services_updated_at, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version AS services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume FROM services 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume WHERE services.deleted = false AND services.`binary` = %(binary_1)s] 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume [parameters: {'binary_1': 'cinder-scheduler'}] 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume (Background on this error at: https://sqlalche.me/e/14/f405) 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume  2023-10-25 07:58:10.440 23 ERROR cinder.cmd.volume [None req-30b4637a-ed28-41c6-974f-7e40d4184954 - - - - - -] No volume service(s) started successfully, terminating. 2023-10-25T07:58:10Z ERROR cannot start service: exited quickly with code 1 ----- unit-cinder-ceph-0: 09:58:10 ERROR unit.cinder-ceph/0.juju-log database:60: Traceback (most recent call last): File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops_sunbeam/guard.py", line 91, in guard yield File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops_sunbeam/charm.py", line 265, in configure_charm self.configure_unit(event) File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops_sunbeam/charm.py", line 477, in configure_unit self.init_container_services() File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/./src/charm.py", line 295, in init_container_services ph.init_service(self.contexts()) File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/./src/charm.py", line 159, in init_service self.start_service() File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/./src/charm.py", line 154, in start_service container.start(self.service_name) File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/model.py", line 1991, in start self._pebble.start_services(service_names) File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/pebble.py", line 1706, in start_services return self._services_action('start', services, timeout, delay) File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/pebble.py", line 1767, in _services_action raise ChangeError(change.err, change) ops.pebble.ChangeError: cannot perform the following tasks: - Start service "cinder-volume" (cannot start service: exited quickly with code 1) ----- Logs from task 0 ----- 2023-10-25T07:58:10Z INFO Most recent service output: (...) 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 548, in query 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 775, in _read_query_result 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume result.read() 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1156, in read 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume first_packet = self.connection._read_packet() 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 725, in _read_packet 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume packet.raise_for_error() 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 221, in raise_for_error 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume err.raise_mysql_exception(self._data) 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume raise errorclass(errno, errval) 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume [SQL: SELECT services.created_at AS services_created_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.uuid AS services_uuid, services.cluster_name AS services_cluster_name, services.host AS services_host, services.`binary` AS services_binary, services.updated_at AS services_updated_at, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version AS services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume FROM services 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume WHERE services.deleted = false AND services.`binary` = %(binary_1)s] 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume [parameters: {'binary_1': 'cinder-scheduler'}] 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume (Background on this error at: https://sqlalche.me/e/14/f405) 2023-10-25 07:58:10.436 23 ERROR cinder.cmd.volume  2023-10-25 07:58:10.440 23 ERROR cinder.cmd.volume [None req-30b4637a-ed28-41c6-974f-7e40d4184954 - - - - - -] No volume service(s) started successfully, terminating. 2023-10-25T07:58:10Z ERROR cannot start service: exited quickly with code 1 ----- unit-cinder-ceph-0: 09:58:11 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 09:59:52 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 10:04:26 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 10:10:05 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 10:14:58 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 10:20:39 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 10:25:27 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 10:29:48 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 10:34:26 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 10:40:14 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 10:44:55 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 10:49:36 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 10:53:44 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 10:59:08 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-cinder-ceph-0: 11:04:45 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)