Cannot deploy locally built charm on arm64 Kubernetes environment

[Claudiu Belu]

A user which owns an arm64 laptop used microk8s to deploy his Kubernetes cluster, and then bootstrapped a Juju controller on it. The controller bootstrapped properly, but when he tried to deploy a locally built charm, it was blocked in Waiting status.

Looking through the Kubernetes cluster reveals that the Charm's pod is stuck in a Pending state. It's status reveals the following:

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-03-30T09:38:55Z"
    message: '0/1 nodes are available: 1 node(s) didn''t match Pod''s node affinity/selector.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending

The pod's nodeSelector is this:

  nodeSelector:
    kubernetes.io/arch: amd64

The pod did not get scheduled on any node because the node selector is set to match amd64 nodes in a cluster that only has arm64.

Looking at the container images used in the Pod Spec (jujusolutions/jujud-operator:2.9.27, jujusolutions/charm-base:ubuntu-20.04), it seems that they are manifest lists which contain images for linux/arm64 [1][2], which means that the Pod should have succeeded in spawning in that environment if the nodeSelector was correct.

Kubernetes pod spec spawned for the charm: https://paste.ubuntu.com/p/yvKwgxJhy2/
Kubernetes nodes: https://paste.ubuntu.com/p/kTMCdgPskR/

Juju version: 2.8.27
Environment: Ubuntu 20.04, linux/arm64.

[1] https://paste.ubuntu.com/p/mcWFBJZM78/
[2] https://paste.ubuntu.com/p/V7MyHPM2N4/

[Heather Lanigan (hmlanigan)]

What happens if you juju deploy with `--constraints arch=amd64`?

Or `juju set-model-constraints arch=amd64` then juju deploy?