cannot bootstrap candidate snap release on kubernetes

[Alex Lutay]

Hi,

Data team is waiting for Juju 3.1.6 bugfixes and tried to use it for Demo (from 3.1/candidate snap channel), but failed.

MicroK8s cannot fetch OCI Image: jujusolutions/jujud-operator:3.1.6.1

The issue:
  Normal Pulling 32s kubelet Pulling image "jujusolutions/jujud-operator:3.1.6.1"
  Warning Failed 1s kubelet Failed to pull image "jujusolutions/jujud-operator:3.1.6.1": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "docker.io/jujusolutions/jujud-operator:3.1.6.1": failed to resolve reference "docker.io/jujusolutions/jujud-operator:3.1.6.1": failed to do request: Head "https://www.docker.com/v2/jujusolutions/jujud-operator/manifests/3.1.6.1": dial tcp 141.193.213.20:443: i/o timeout
  Warning Failed 1s kubelet Error: ErrImagePull
  Normal BackOff 0s kubelet Back-off pulling image "jujusolutions/jujud-operator:3.1.6.1"
  Warning Failed 0s kubelet Error: ImagePullBackOff

The full K8s log: https://pastebin.canonical.com/p/fHpBrW66qS/
Reason: 3.1.6.1 is not available here
https://streams.canonical.com/juju/tools/streams/v1/com.ubuntu.juju-released-agents.sjson

The bootstrap with --agent-version 3.1.6 doesn't help either.

HUGE thanks to Pedro Guimarães for noticing this!

P.S. juju bootstrap with debug: https://pastebin.canonical.com/p/RWWBxydrjV/
...
09:45:35 INFO juju.environs.bootstrap tools.go:78 looking for bootstrap agent binaries: version=3.1.6
09:45:35 DEBUG juju.environs.tools tools.go:87 finding agent binaries in stream: "released"
09:45:35 DEBUG juju.environs.tools tools.go:89 reading agent binaries with major.minor version 3.1
09:45:35 DEBUG juju.environs.tools tools.go:98 filtering agent binaries by version: 3.1.6
...
09:45:36 DEBUG juju.environs.tools build.go:364 forcing version to 3.1.6.1

[Alex Lutay (taurus)]

The workaround (from Pedro) is to choose stream 'proposed':

juju bootstrap
--config container-image-stream=proposed \
--config agent-stream=proposed \
--agent-version 3.1.6 \
mycloud mycontroller

I would ask for the fix here, as candidate risk should work out of the box, IMHO. Tnx!

[Harry Pidcock (hpidcock)]

We'll look at this for future releases, probably in the 4.x series as it will require some changes to our build/release process.

[Ian Booth (wallyworld)]

The Juju snap provides the Juju *client*. The Juju client is managed separately from the Juju *agent*. A Juju client can bootstrap with any number of agent versions. Just as snap provides a channel to specify what risk to use when asking for the Juju client, so too does the agent metadata allow for a risk to be specified when requesting an agent. Candidate releases of Juju are not stable and so must be opt in, just as candidate snaps are opt in. The Juju agents are cataloged using simplestreams metadata, and the way you specify what risk you want is to use the "agent-stream" config. This is orthogonal to the risk of the snap Juju client. A 3.1.6 candidate client will happily bootstrap a released 3.1.5 controller without any additional args, but if you want to bootstrap a non-released, 3.1.6 candidate controller, you must opt in.

Note the container-image-stream is not needed, that's for lxd/kvm images from memory. You don't even need the agent-version param, just do

juju bootstrap --config agent-stream=proposed mycloud

This is no different from

snap install mysnap --channel=candidate

In the above snap case, "snap" is the cli client (like "juju"), and the "mysnap" is the workload (like the juju controller is).

[Ian Booth (wallyworld)]

I can see how this can be confusing though. So we need better messaging perhaps.