metallb-controller errors with container error: "exec /controller: exec format error"

[Bas de Bruijne]

In test run https://solutions.qa.canonical.com/v2/testruns/9f9a5487-4066-4f6f-b091-45826bc19de8/, the metallb-controller dies with the following message:

=====================
Model Controller Cloud/Region Version SLA Timestamp
metallb-system foundations-maas kubernetes_cloud/default 2.9.42 unsupported 21:34:48Z

App Version Status Scale Charm Channel Rev Address Exposed Message
metallb-controller res:metallb-controller-imag... waiting 1 metallb-controller stable 41 10.152.183.127 no
metallb-speaker res:metallb-speaker-image@6... waiting 6/8 metallb-speaker stable 36 10.152.183.218 no

Unit Workload Agent Address Ports Message
metallb-controller/0* error idle 192.168.255.195 7472/TCP container error:
metallb-speaker/0* waiting idle 10.246.201.62 7472/TCP waiting for container
metallb-speaker/1 waiting executing 10.246.201.8 7472/TCP (install) waiting for container
metallb-speaker/2 maintenance executing 10.246.201.64 7472/TCP (install) installing charm software
metallb-speaker/3 waiting allocating 10.246.201.7 7472/TCP waiting for container
metallb-speaker/4 waiting executing 10.246.201.60 7472/TCP (install) waiting for container
metallb-speaker/5 waiting executing 10.246.201.5 7472/TCP (install) waiting for container
metallb-speaker/6 waiting allocating 10.246.201.65 7472/TCP waiting for container
metallb-speaker/7 waiting executing 10.246.201.63 7472/TCP (install) waiting for container
=====================

In the pod logs we see:
=====================
$ cat 6/baremetal/pod-logs/metallb-system-metallb-controller-57df8d7b5c-h5zbf-controller.log
exec /controller: exec format error
=====================

I can't find any indicators of why this is happening.

Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/9f9a5487-4066-4f6f-b091-45826bc19de8/index.html

[Stone Preston (stonepreston)]

Generally this indicates a mismatch between container image architectures and the arch of the worker machine. What arch are the workers?

[Stone Preston (stonepreston)]

This should have a similar workaround to https://bugs.launchpad.net/charm-coredns/+bug/1998607

The charm oci image registry does not allow for multiarch images, so the image resource associated with the charm is amd64 only. You can workaround this by manually specifying the metallb image we have in rocks which is a multiarch image:

$ cat <<EOF > metallb-speaker.json
{"ImageName": "rocks.canonical.com:443/cdk/metallb/speaker:v0.12"}
EOF

juju attach-resource metallb-speaker --metallb-speaker-image=./metallb-speaker.json

and do the same for the controller charm as well (the image url for the controller is rocks.canonical.com:443/cdk/metallb/controller:v0.12

[Bas de Bruijne (basdbruijne)]

This is running on arm64, so that is probably the issue

[Adam Dyess (addyess)]

Beginning in 1.28 track of this charm, the metallb image is no longer a charm resource. This means that this charm will not encounter binary specific issues as it will direct the kubernetes cluster to fetch its images from rocks.canonical.com (or configured image-registry) appropriately based on the machine architecture scheduled to run the speakers and controllers.