Image pull errors when trying to deploy bundle
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Fix Released
|
Medium
|
Unassigned |
Bug Description
Issue description
We are getting image pull errors when trying to deploy a charm bundle.
When looking at k8s log, we get the following message:
```
"jujusolutions/
```
Recreation steps
- On a VM running ubuntu 20.04, install microk8s and Juju. Then run `juju deploy magma-orc8r`.
Reference
- bundle source: https:/
Setup
- Ubuntu 20.04
- Microk8s v1.22.4
- Juju
Logs
ubuntu@
Model Controller Cloud/Region Version SLA Timestamp
dev9 microk8s-localhost microk8s/localhost 2.9.21 unsupported 00:19:40Z
App Version Status Scale Charm Store Channel Rev OS Address Message
nms-magmalte waiting 0/1 magma-nms-magmalte charmhub edge 3 kubernetes 10.152.183.21 installing agent
nms-nginx-proxy waiting 0/1 magma-nms-
orc8r-accessd waiting 0/1 magma-orc8r-accessd charmhub edge 5 kubernetes 10.152.183.63 installing agent
orc8r-analytics waiting 0/1 magma-orc8r-
orc8r-bootstrapper waiting 0/1 magma-orc8r-
orc8r-certifier waiting 0/1 magma-orc8r-
orc8r-configurator waiting 0/1 magma-orc8r-
orc8r-ctraced waiting 0/1 magma-orc8r-ctraced charmhub edge 1 kubernetes 10.152.183.2 installing agent
orc8r-device waiting 0/1 magma-orc8r-device charmhub edge 1 kubernetes 10.152.183.149 installing agent
orc8r-directoryd waiting 0/1 magma-orc8r-
orc8r-dispatcher waiting 0/1 magma-orc8r-
orc8r-eventd waiting 0/1 magma-orc8r-eventd charmhub edge 1 kubernetes 10.152.183.14 installing agent
orc8r-ha waiting 0/1 magma-orc8r-ha charmhub edge 1 kubernetes 10.152.183.97 installing agent
orc8r-lte waiting 0/1 magma-orc8r-lte charmhub edge 1 kubernetes 10.152.183.79 installing agent
orc8r-metricsd waiting 0/1 magma-orc8r-
orc8r-nginx waiting 0/1 magma-orc8r-nginx charmhub edge 1 kubernetes 10.152.183.129 installing agent
orc8r-obsidian waiting 0/1 magma-orc8r-
orc8r-orchestrator waiting 0/1 magma-orc8r-
orc8r-policydb waiting 0/1 magma-orc8r-
orc8r-service-
orc8r-smsd waiting 0/1 magma-orc8r-smsd charmhub edge 1 kubernetes 10.152.183.70 installing agent
orc8r-state waiting 0/1 magma-orc8r-state charmhub edge 1 kubernetes 10.152.183.244 installing agent
orc8r-streamer waiting 0/1 magma-orc8r-
orc8r-subscriberdb waiting 0/1 magma-orc8r-
orc8r-subscribe
orc8r-tenants waiting 0/1 magma-orc8r-tenants charmhub edge 1 kubernetes 10.152.183.91 installing agent
postgresql-k8s .../postgresql@
Unit Workload Agent Address Ports Message
nms-magmalte/0 waiting allocating 10.1.218.31 installing agent
nms-nginx-proxy/0 waiting allocating 10.1.218.44 installing agent
orc8r-accessd/0 waiting allocating 10.1.218.27 installing agent
orc8r-analytics/0 waiting allocating 10.1.218.14 installing agent
orc8r-bootstrap
orc8r-certifier/0 waiting allocating 10.1.218.28 installing agent
orc8r-configura
orc8r-ctraced/0 waiting allocating 10.1.218.32 installing agent
orc8r-device/0 waiting allocating 10.1.218.47 installing agent
orc8r-directoryd/0 waiting allocating 10.1.218.35 installing agent
orc8r-dispatcher/0 waiting allocating 10.1.218.37 installing agent
orc8r-eventd/0 waiting allocating 10.1.218.26 installing agent
orc8r-ha/0 waiting allocating 10.1.218.33 installing agent
orc8r-lte/0 waiting allocating 10.1.218.39 installing agent
orc8r-metricsd/0 waiting allocating 10.1.218.24 installing agent
orc8r-nginx/0 waiting allocating 10.1.218.41 installing agent
orc8r-obsidian/0 waiting allocating 10.1.218.30 installing agent
orc8r-orchestra
orc8r-policydb/0 waiting allocating 10.1.218.13 installing agent
orc8r-service-
orc8r-smsd/0 waiting allocating 10.1.218.18 installing agent
orc8r-state/0 waiting allocating 10.1.218.58 installing agent
orc8r-streamer/0 waiting allocating 10.1.218.61 installing agent
orc8r-subscribe
orc8r-subscribe
orc8r-tenants/0 waiting allocating 10.1.218.8 installing agent
postgresql-k8s/0* active idle 10.1.218.64 5432/TCP Pod configured
ubuntu@
NAME READY STATUS RESTARTS AGE
modeloperator-
postgresql-
postgresql-k8s-0 1/1 Running 0 18m
orc8r-device-0 1/2 ImagePullBackOff 0 21m
orc8r-ctraced-0 1/2 ImagePullBackOff 0 21m
orc8r-dispatcher-0 1/2 ImagePullBackOff 0 20m
orc8r-directoryd-0 1/2 ImagePullBackOff 0 21m
orc8r-eventd-0 1/2 ImagePullBackOff 0 20m
orc8r-metricsd-0 1/2 ImagePullBackOff 0 20m
orc8r-lte-0 1/2 ImagePullBackOff 0 20m
orc8r-ha-0 1/2 ImagePullBackOff 0 20m
orc8r-obsidian-0 1/2 ImagePullBackOff 0 20m
orc8r-orchestra
orc8r-service-
orc8r-state-0 1/2 ImagePullBackOff 0 19m
orc8r-smsd-0 1/2 ImagePullBackOff 0 19m
orc8r-policydb-0 1/2 ImagePullBackOff 0 20m
orc8r-nginx-0 1/2 ImagePullBackOff 0 20m
orc8r-streamer-0 1/2 ImagePullBackOff 0 19m
orc8r-subscribe
orc8r-tenants-0 1/2 ImagePullBackOff 0 19m
orc8r-subscribe
nms-magmalte-0 1/2 ImagePullBackOff 0 23m
nms-nginx-proxy-0 1/2 ImagePullBackOff 0 22m
orc8r-accessd-0 1/2 ImagePullBackOff 0 22m
orc8r-analytics-0 1/2 ImagePullBackOff 0 22m
orc8r-bootstrap
orc8r-configura
orc8r-certifier-0 1/2 ImagePullBackOff 0 21m
ubuntu@
Name: orc8r-subscribe
Namespace: dev9
Priority: 0
Node: ip-172-
Start Time: Thu, 09 Dec 2021 00:00:11 +0000
Labels: app.kubernetes.
Annotations: cni.projectcali
Status: Pending
IP: 10.1.218.59
IPs:
IP: 10.1.218.59
Controlled By: StatefulSet/
Init Containers:
charm-init:
Container ID: containerd:
Image: jujusolutions/
Image ID: sha256:
Port: <none>
Host Port: <none>
Command:
/
Args:
init
--data-dir
/var/lib/juju
--bin-dir
/charm/bin
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 09 Dec 2021 00:00:12 +0000
Finished: Thu, 09 Dec 2021 00:00:12 +0000
Ready: True
Restart Count: 0
Environment Variables from:
orc8r-
Environment:
JUJU_
JUJU_
JUJU_
Mounts:
/charm/bin from charm-data (rw,path=
/
/var/lib/juju from charm-data (rw,path=
/
Containers:
charm:
Container ID:
Image: jujusolutions/
Image ID:
Port: <none>
Host Port: <none>
Command:
/
Args:
unit
--data-dir
/var/lib/juju
-
0
--append-env
PATH=
--show-log
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Liveness: http-get http://
Readiness: http-get http://
Startup: http-get http://
Environment:
JUJU_
HTTP_
Mounts:
/charm/bin from charm-data (ro,path=
/
/var/lib/juju from charm-data (rw,path=
/
magma-
Container ID: containerd:
Image: registry.
Image ID: registry.
Port: <none>
Host Port: <none>
Command:
/
Args:
run
--create-dirs
--hold
--verbose
State: Running
Started: Thu, 09 Dec 2021 00:00:14 +0000
Ready: True
Restart Count: 0
Environment:
JUJU_
PEBBLE_
Mounts:
/
/
/
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
charm-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpira
ConfigMapName: kube-root-ca.crt
ConfigMapOp
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 19m default-scheduler Successfully assigned dev9/orc8r-
Normal Pulled 19m kubelet Container image "jujusolutions/
Normal Created 19m kubelet Created container charm-init
Normal Started 19m kubelet Started container charm-init
Normal Pulled 19m kubelet Container image "registry.
Normal Created 19m kubelet Created container magma-orc8r-
Normal Started 19m kubelet Started container magma-orc8r-
Normal Pulling 18m (x3 over 19m) kubelet Pulling image "jujusolutions/
Warning Failed 18m (x3 over 19m) kubelet Failed to pull image "jujusolutions/
Warning Failed 18m (x3 over 19m) kubelet Error: ErrImagePull
Warning Failed 17m (x5 over 19m) kubelet Error: ImagePullBackOff
Normal BackOff 3m50s (x66 over 19m) kubelet Back-off pulling image "jujusolutions/
bundle.yaml
bundle: kubernetes
name: magma-orc8r
description: |
Orchestrator is a Magma service that provides a simple and consistent way to
configure and monitor the wireless network securely. The metrics acquired through the platform
allows you to see the analytics and traffic flows of the wireless users through the Magma web UI.
applications:
nms-magmalte:
charm: magma-nms-magmalte
series: kubernetes
channel: edge
revision: 3
resources:
magma-
scale: 1
nms-nginx-proxy:
charm: magma-nms-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-accessd:
charm: magma-orc8r-accessd
series: kubernetes
channel: edge
revision: 5
resources:
magma-
scale: 1
orc8r-analytics:
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-certifier:
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
options:
domain: example.com
orc8r-
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-ctraced:
charm: magma-orc8r-ctraced
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-device:
charm: magma-orc8r-device
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-directoryd:
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-dispatcher:
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-eventd:
charm: magma-orc8r-eventd
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-ha:
charm: magma-orc8r-ha
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-lte:
charm: magma-orc8r-lte
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-metricsd:
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-nginx:
charm: magma-orc8r-nginx
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-obsidian:
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-policydb:
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-smsd:
charm: magma-orc8r-smsd
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-state:
charm: magma-orc8r-state
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-streamer:
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-
charm: magma-orc8r-
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
orc8r-tenants:
charm: magma-orc8r-tenants
series: kubernetes
channel: edge
revision: 1
resources:
magma-
scale: 1
postgresql-k8s:
charm: postgresql-k8s
series: kubernetes
channel: stable
revision: 3
resources:
postgresq
scale: 1
relations:
- - nms-magmalte:
- orc8r-certifier
- - nms-nginx-
- orc8r-certifier
- - orc8r-accessd:db
- postgresql-k8s:db
- - orc8r-bootstrap
- orc8r-certifier
- - orc8r-certifier:db
- postgresql-k8s:db
- - orc8r-configura
- postgresql-k8s:db
- - orc8r-ctraced:db
- postgresql-k8s:db
- - orc8r-device:db
- postgresql-k8s:db
- - orc8r-directoryd:db
- postgresql-k8s:db
- - orc8r-lte:db
- postgresql-k8s:db
- - orc8r-policydb:db
- postgresql-k8s:db
- - orc8r-smsd:db
- postgresql-k8s:db
- - orc8r-state:db
- postgresql-k8s:db
- - orc8r-subscribe
- postgresql-k8s:db
- - orc8r-subscribe
- postgresql-k8s:db
- - orc8r-tenants:db
- postgresql-k8s:db
- - orc8r-nginx:
- orc8r-certifier
- - orc8r-nginx:
- orc8r-bootstrap
- - orc8r-nginx:
- orc8r-obsidian:
- - nms-magmalte:db
- postgresql-k8s:db
- - nms-nginx-
- nms-magmalte:
Changed in juju: | |
milestone: | 2.9.22 → 2.9.23 |
Changed in juju: | |
milestone: | 2.9.23 → 2.9.24 |
Changed in juju: | |
milestone: | 2.9.24 → 2.9.25 |
Changed in juju: | |
milestone: | 2.9.25 → 2.9.26 |
Changed in juju: | |
importance: | High → Medium |
milestone: | 2.9.26 → none |
Confirmed deploying the bundle results in many ImagePullBackOff and ErrImagePull. However this does not occur if you deploy all the charms by "hand", outside of a bundle.
I also tried removing the revision and resource parts for each charm from the bundle with the same result.