Charm fails to go ready in juju 3.5

[Adam Dyess]

Uncaught exception found in charm code during coredns-pebble-ready and the charm never resolves

unit-coredns-0: 2024-06-22 14:21:38 ERROR unit.coredns/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-coredns-0/charm/./src/charm.py", line 171, in <module>
    main(CoreDNSCharm)
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/main.py", line 544, in main
    manager.run()
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/main.py", line 520, in run
    self._emit()
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/main.py", line 509, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/main.py", line 143, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/framework.py", line 352, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/framework.py", line 851, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/framework.py", line 941, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-coredns-0/charm/./src/charm.py", line 64, in _on_coredns_pebble_ready
    self._push_corefile_config(event)
  File "/var/lib/juju/agents/unit-coredns-0/charm/./src/charm.py", line 143, in _push_corefile_config
    container.push("/etc/coredns/Corefile", corefile, make_dirs=True)
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/model.py", line 2302, in push
    self._pebble.push(str(path), source, encoding=encoding,
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/pebble.py", line 2256, in push
    self._raise_on_path_error(typing.cast('_FilesResponse', resp), path)
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/pebble.py", line 2204, in _raise_on_path_error
    raise PathError(error['kind'], error['message'])
ops.pebble.PathError: permission-denied - cannot create directory: mkdir /etc/coredns: permission denied

[Adam Dyess (addyess)]

Observed on 1.29/stable (rev 141) of the charm

[Adam Dyess (addyess)]

It occurred 4 times in CI
https://jenkaas.s3.amazonaws.com/619534e7-dfa3-4ce1-ae43-1fb4cfcf0a0c/index.html
https://jenkaas.s3.amazonaws.com/049ecb4a-ea6d-40d3-bf18-13e8025d323d/index.html
https://jenkaas.s3.amazonaws.com/3aa56aa1-3066-44f3-9d7c-4a43556f5cab/index.html
https://jenkaas.s3.amazonaws.com/4c4b3a22-1f7a-44b0-afe0-bd01d79a92d9/index.html

But i've not had success reproducing locally