OpenStack Snap

resize fails on cinder-ceph hook failed: "storage-backend-relation-changed"

Bug #2045031 reported by Marian Gasparovic on 2023-11-28

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Snap	Triaged	Low	Unassigned

Bug Description

During sunbeam resize cinder-ceph gopes to error state with hook failed: "storage-backend-relation-changed"

Logs show a traceback

2023-11-26T05:18:41.196Z [container-agent] 2023-11-26 05:18:41 ERROR juju-log storage-backend:89: Uncaught exception while in charm code:
2023-11-26T05:18:41.196Z [container-agent] Traceback (most recent call last):
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/./src/charm.py", line 463, in <module>
2023-11-26T05:18:41.196Z [container-agent] main(CinderCephOperatorCharm)
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/main.py", line 436, in main
2023-11-26T05:18:41.196Z [container-agent] _emit_charm_event(charm, dispatcher.event_name)
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/main.py", line 144, in _emit_charm_event
2023-11-26T05:18:41.196Z [container-agent] event_to_emit.emit(*args, **kwargs)
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/framework.py", line 340, in emit
2023-11-26T05:18:41.196Z [container-agent] framework._emit(event)
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/framework.py", line 842, in _emit
2023-11-26T05:18:41.196Z [container-agent] self._reemit(event_path)
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/framework.py", line 931, in _reemit
2023-11-26T05:18:41.196Z [container-agent] custom_handler(event)
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/lib/charms/cinder_k8s/v0/storage_backend.py", line 184, in _on_storage_backend_relation_changed
2023-11-26T05:18:41.196Z [container-agent] self.on.api_ready.emit()
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/framework.py", line 340, in emit
2023-11-26T05:18:41.196Z [container-agent] framework._emit(event)
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/framework.py", line 842, in _emit
2023-11-26T05:18:41.196Z [container-agent] self._reemit(event_path)
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/framework.py", line 931, in _reemit
2023-11-26T05:18:41.196Z [container-agent] custom_handler(event)
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/./src/charm.py", line 114, in _on_ready
2023-11-26T05:18:41.196Z [container-agent] self.callback_f(event)
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/./src/charm.py", line 275, in api_ready
2023-11-26T05:18:41.196Z [container-agent] handler.start_service()
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/./src/charm.py", line 191, in start_service
2023-11-26T05:18:41.196Z [container-agent] service = container.get_service(self.service_name)
2023-11-26T05:18:41.196Z [container-agent] File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/model.py", line 2182, in get_service
2023-11-26T05:18:41.196Z [container-agent] raise ModelError(f'service {service_name!r} not found')
2023-11-26T05:18:41.196Z [container-agent] ops.model.ModelError: service 'cinder-volume' not found

Tags:

Revision history for this message

James Page (james-page) wrote on 2023-11-29:

Download full text (4.9 KiB)

This is the cause I think:

This is the cause I think:

2023-11-26T03:51:42.027Z [container-agent] 2023-11-26 03:51:42 WARNING juju-log storage-backend:89: Charm is waiting in section 'Bootstrapping' due to 'Container service not ready'
2023-11-26T03:51:42.414Z [container-agent] 2023-11-26 03:51:42 ERROR juju-log storage-backend:89: Uncaught exception while in charm code:
2023-11-26T03:51:42.414Z [container-agent] Traceback (most recent call last):
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/./src/charm.py", line 463, in <module>
2023-11-26T03:51:42.414Z [container-agent]     main(CinderCephOperatorCharm)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/main.py", line 436, in main
2023-11-26T03:51:42.414Z [container-agent]     _emit_charm_event(charm, dispatcher.event_name)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/main.py", line 144, in _emit_charm_event
2023-11-26T03:51:42.414Z [container-agent]     event_to_emit.emit(*args, **kwargs)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/framework.py", line 340, in emit
2023-11-26T03:51:42.414Z [container-agent]     framework._emit(event)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/framework.py", line 842, in _emit
2023-11-26T03:51:42.414Z [container-agent]     self._reemit(event_path)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/framework.py", line 931, in _reemit
2023-11-26T03:51:42.414Z [container-agent]     custom_handler(event)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/lib/charms/cinder_k8s/v0/storage_backend.py", line 184, in _on_storage_backend_relation_changed
2023-11-26T03:51:42.414Z [container-agent]     self.on.api_ready.emit()
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/framework.py", line 340, in emit
2023-11-26T03:51:42.414Z [container-agent]     framework._emit(event)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/framework.py", line 842, in _emit
2023-11-26T03:51:42.414Z [container-agent]     self._reemit(event_path)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/framework.py", line 931, in _reemit
2023-11-26T03:51:42.414Z [container-agent]     custom_handler(event)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/./src/charm.py", line 114, in _on_ready
2023-11-26T03:51:42.414Z [container-agent]     self.callback_f(event)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/./src/charm.py", line 275, in api_ready
2023-11-26T03:51:42.414Z [container-agent]     handler.start_service()
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/./src/charm.py", line 191, in start_service
2023-11-26T03:51:42.414Z [container-agent]     service = container.get_service(self.service_name)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/model.py", line 2180, in get_service
2023-11-26T03:51:42.414Z [container-agent]     services = self.get_services(service_name)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/model.py", line 2171, in get_services
2023-11-26T03:51:42.414Z [container-agent]     services = self._pebble.get_services(names)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/pebble.py", line 1931, in get_services
2023-11-26T03:51:42.414Z [container-agent]     resp = self._request('GET', '/v1/services', query)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/pebble.py", line 1588, in _request
2023-11-26T03:51:42.414Z [container-agent]     response = self._request_raw(method, path, query, headers, data)
2023-11-26T03:51:42.414Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-ceph-0/charm/venv/ops/pebble.py", line 1637, in _request_raw
2023-11-26T03:51:42.414Z [container-agent]     raise ConnectionError(
2023-11-26T03:51:42.414Z [container-agent] ops.pebble.ConnectionError: Could not connect to Pebble: socket not found at '/charm/containers/cinder-volume/pebble.socket' (container restarted?)

Looking at the deployment the cinder-volume container seems to have got restarted and then the pebble-ready event never re-fired.

The code in the original bug report does not guard against this particular condition so could be more robust but we need to understand a) why the container got restarted and b) why the pebble ready event did not refire, causing the layers that define the service to get re-created.

Revision history for this message

James Page (james-page) wrote on 2023-11-29:

https://oil-jenkins.canonical.com/artifacts/4d0afc99-3abb-4c2d-a138-15216375c64b/index.html

James Page (james-page) on 2023-12-05

Changed in snap-openstack:
importance:	Undecided → Low
status:	New → Confirmed
status:	Confirmed → Triaged

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.