ceph-mon fault in osd-relation Object of type bytes is not JSON serializable

[Peter Sabaini]

Got a fault in CI where ceph-fs tries to create a pool two times, which is not properly handled by ceph-mon, see below traceback

CI artefacts:
https://openstack-ci-reports.ubuntu.com/artifacts/795/896998/1/check/jammy-bobcat/795fc26/

2023-10-02 09:04:21.540624 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: 2023-10-02 09:04:18 WARNING unit.ceph-mon/0.juju-log server.go:316 osd:2: DEPRECATION WARNING: Function one_shot_log is being removed : Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
2023-10-02 09:04:21.540946 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: 2023-10-02 09:04:19 WARNING unit.ceph-mon/0.osd-relation-changed logger.go:60 filesystem 'ceph-fs' already exists
2023-10-02 09:04:21.540968 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: 2023-10-02 09:04:20 WARNING unit.ceph-mon/0.osd-relation-changed logger.go:60 Error EINVAL: RADOS pool 'ec_ceph-fs_data' is already used by filesystem 'ceph-fs' as a 'data' pool for application 'cephfs'
2023-10-02 09:04:21.540977 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: 2023-10-02 09:04:20 ERROR unit.ceph-mon/0.juju-log server.go:316 osd:2: b''
2023-10-02 09:04:21.540983 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: 2023-10-02 09:04:20 ERROR unit.ceph-mon/0.juju-log server.go:316 osd:2: Uncaught exception while in charm code:
2023-10-02 09:04:21.540989 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: Traceback (most recent call last):
2023-10-02 09:04:21.541012 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/./src/charm.py", line 310, in <module>
2023-10-02 09:04:21.541019 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: main(CephMonCharm)
2023-10-02 09:04:21.541040 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/ops/main.py", line 441, in main
2023-10-02 09:04:21.541257 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: _emit_charm_event(charm, dispatcher.event_name)
2023-10-02 09:04:21.541279 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
2023-10-02 09:04:21.541286 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: event_to_emit.emit(*args, **kwargs)
2023-10-02 09:04:21.541292 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/ops/framework.py", line 342, in emit
2023-10-02 09:04:21.541299 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: framework._emit(event)
2023-10-02 09:04:21.541305 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/ops/framework.py", line 839, in _emit
2023-10-02 09:04:21.541311 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: self._reemit(event_path)
2023-10-02 09:04:21.541317 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/ops/framework.py", line 928, in _reemit
2023-10-02 09:04:21.541323 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: custom_handler(event)
2023-10-02 09:04:21.541365 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/./src/charm.py", line 139, in on_osd_relation
2023-10-02 09:04:21.541376 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: self.on.notify_clients.emit()
2023-10-02 09:04:21.541383 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/ops/framework.py", line 342, in emit
2023-10-02 09:04:21.541389 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: framework._emit(event)
2023-10-02 09:04:21.541395 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/ops/framework.py", line 839, in _emit
2023-10-02 09:04:21.541401 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: self._reemit(event_path)
2023-10-02 09:04:21.541407 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/ops/framework.py", line 928, in _reemit
2023-10-02 09:04:21.541412 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: custom_handler(event)
2023-10-02 09:04:21.541418 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/./src/charm.py", line 193, in notify_clients
2023-10-02 09:04:21.541778 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: self.mds.notify_all()
2023-10-02 09:04:21.541799 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/src/ceph_client.py", line 61, in notify_all
2023-10-02 09:04:21.541806 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: self._handle_client_relation(relation, unit)
2023-10-02 09:04:21.541813 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/src/ceph_mds.py", line 66, in _handle_client_relation
2023-10-02 09:04:21.541819 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: super()._handle_client_relation(relation, unit)
2023-10-02 09:04:21.541840 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/src/ceph_client.py", line 122, in _handle_client_relation
2023-10-02 09:04:21.541847 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: self._handle_broker_request(
2023-10-02 09:04:21.541853 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/src/ceph_client.py", line 193, in _handle_broker_request
2023-10-02 09:04:21.541859 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: rsp = self.charm.process_broker_request(
2023-10-02 09:04:21.542098 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/./src/charm.py", line 289, in process_broker_request
2023-10-02 09:04:21.542140 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: broker_result = process_requests(requests)
2023-10-02 09:04:21.542148 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/charms_ceph/broker.py", line 109, in decode_inner
2023-10-02 09:04:21.542154 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: return json.dumps(f(json.loads(req)))
2023-10-02 09:04:21.542160 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/usr/lib/python3.10/json/__init__.py", line 231, in dumps
2023-10-02 09:04:21.542166 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: return _default_encoder.encode(obj)
2023-10-02 09:04:21.542172 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
2023-10-02 09:04:21.542178 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: chunks = self.iterencode(o, _one_shot=True)
2023-10-02 09:04:21.542184 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
2023-10-02 09:04:21.542383 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: return _iterencode(o, 0)
2023-10-02 09:04:21.542417 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: File "/usr/lib/python3.10/json/encoder.py", line 179, in default
2023-10-02 09:04:21.542426 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: raise TypeError(f'Object of type {o.__class__.__name__} '
2023-10-02 09:04:21.542441 | focal-medium | 2023-10-02 09:04:21 [ERROR] unit-ceph-mon-0.log: TypeError: Object of type bytes is not JSON serializable