fork/exec error when deploying k8s charm to arm64

[Marcelo Henrique Neppel]

Hi Juju team!

I was investigating one issue from a member of the community and to understand better the deployment of charms on arm64 (M2 proc) I tried to deploy two charms that have revisions for arm64:
- https://charmhub.io/kubernetes-dashboard?channel=1.28/edge
- https://charmhub.io/coredns

However, both failed with the following error:

unit-coredns-0: 14:38:08 ERROR unit.coredns/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-coredns-0/charm/./src/charm.py", line 182, in <module>
    main(CoreDNSCharm)
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/framework.py", line 354, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/framework.py", line 830, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/framework.py", line 919, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-coredns-0/charm/./src/charm.py", line 74, in _on_coredns_pebble_ready
    container.autostart()
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/model.py", line 1874, in autostart
    self._pebble.autostart_services()
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/pebble.py", line 1561, in autostart_services
    return self._services_action('autostart', [], timeout, delay)
  File "/var/lib/juju/agents/unit-coredns-0/charm/venv/ops/pebble.py", line 1659, in _services_action
    raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "coredns" (cannot start service: fork/exec /coredns: exec format error)
----- Logs from task 0 -----
2023-06-28T17:38:08Z ERROR cannot start service: fork/exec /coredns: exec format error
-----
unit-coredns-0: 14:38:09 ERROR juju.worker.uniter.operation hook "coredns-pebble-ready" (via hook dispatching script: dispatch) failed: exit status 1
unit-coredns-0: 14:38:09 ERROR juju.worker.uniter pebble poll failed for container "coredns": failed to send pebble-ready event: hook failed

Those charms worked fine on amd64 on a cloud instance (at least I saw that they reached the active/idle state).

Is is a problem in the charms or in Juju?

[Marcelo Henrique Neppel (neppel)]

OS: Ubuntu 22.04 running through Multipass on macOS 13.4.1
Juju version: 2.9.43
microk8s version: v1.24.14

[Joseph Phillips (manadart)]

This looks to be emitted from Pebble.

Not ruling out juju, but I think we'd fail earlier if, for example we had laid down a binary for an incorrect architecture.

I can't find a LP project for Pebble, so I'm not sure what it should be against, but I've assigned to Ben for any insights he can provide.

[Ben Hoyt (benhoyt)]

This looks like a charm packaging / architecture issue. Pebble is saying it can't start the "/coredns" executable, because it's the wrong architecture (that's the "exec format error"). Whatever is putting that executable there must be getting the arch wrong: I'd guess it's an amd64 binary that the charm is trying to run on arm64 (or vice versa?). I think the binary comes from a charm resource. So I suspect the charm was packaged or uploaded incorrectly, with the wrong architecture attached as a resource. Could be a charmcraft bug, or (more likely) how charmcraft was used. Looks like Cory Johns is the charm maintainer, and Adam Dyess the most recent committer to the charm repo.

[Marcelo Henrique Neppel (neppel)]

Thanks for the details! I'm gonna talk to the team that developed those charms.

[Marcelo Henrique Neppel (neppel)]

Just to update this issue, I confirmed the there are issues with the architecture of the binaries inside the images of the charms that I commented. For coredns, for example, there is a workaround in the description of https://bugs.launchpad.net/charm-coredns/+bug/1998607. Thanks for the help!