Brief Description
-----------------
Apply any application vault,portieris , the pods fails with "ImagePullBackOff" error since the
kubelet fails to resolve reference "registry.local:9001/docker.io/hashicorp/vault-k8s:0.4.0""
Severity
--------
Major
Steps to Reproduce
------------------
1)Upload vault and apply
system application-upload /usr/local/share/applications/helm/vault-20.06-14.tgz
system application-apply vault
2)Now monitor the pods status as it fails to download the image form local registry
[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
sva-vault-0 0/1 ImagePullBackOff 0 10m
sva-vault-agent-injector-db6878c69-v7ghr 0/1 ImagePullBackOff 0 10m
sva-vault-manager-0 1/1 Running 0 10m
[sysadmin@controller-0 ~(keystone_admin)]$
3)pod events
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 11m (x4 over 11m) default-scheduler running "VolumeBinding" filter plugin for pod "sva-vault-0": pod has unbound immediate PersistentVolumeClaims
Normal Scheduled 10m default-scheduler Successfully assigned vault/sva-vault-0 to controller-0
Normal SuccessfulAttachVolume 10m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-26224005-38c7-4dc6-ae78-7d941867b161"
Normal SuccessfulAttachVolume 10m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-ede416a5-6708-4d52-b4f6-3e95f37d79a3"
Normal Pulling 8m56s (x4 over 10m) kubelet, controller-0 Pulling image "registry.local:9001/docker.io/vault:1.4.2"
Warning Failed 8m55s (x4 over 10m) kubelet, controller-0 Failed to pull image "registry.local:9001/docker.io/vault:1.4.2": rpc error: code = Unknown desc = failed to pull and unpack image "registry.local:9001/docker.io/vault:1.4.2": failed to resolve reference "registry.local:9001/docker.io/vault:1.4.2": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Warning Failed 8m55s (x4 over 10m) kubelet, controller-0 Error: ErrImagePull
Warning Failed 8m29s (x7 over 10m) kubelet, controller-0 Error: ImagePullBackOff
Normal BackOff 22s (x42 over 10m) kubelet, controller-0 Back-off pulling image "registry.local:9001/docker.io/vault:1.4.2"
4)Same issue for portieris images, i think it happens for any image from the local registry.
5)Also manual pull fails with "crictl"
[sysadmin@controller-0 ~(keystone_admin)]$ crictl pull registry.local:9001/docker.io/vault:1.4.2
FATA[0000] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry.local:9001/docker.io/vault:1.4.2": failed to resolve reference "registry.local:9001/docker.io/vault:1.4.2": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
but works with docker pull
[sysadmin@controller-0 ~(keystone_admin)]$ sudo docker pull registry.local:9001/docker.io/vault:1.4.2
Password:
1.4.2: Pulling from vault
21c83c524219: Pull complete
d710f8cd918f: Pull complete
fd8d5779418b: Extracting [==================> ] 17.83MB/49.32MB
a42e38f40189: Download complete
95183e98f2b7: Download complete
Expected Behavior
------------------
Images should be successfully downloaded from local registry
Actual Behavior
----------------
Images fail to download from the local registry
Reproducibility
---------------
Reproducible
System Configuration
--------------------
standard controller duplex
Branch/Pull Time/Commit
-----------------------
master approximately 2021-01-13
Brief Description local:9001/ docker. io/hashicorp/ vault-k8s: 0.4.0""
-----------------
Apply any application vault,portieris , the pods fails with "ImagePullBackOff" error since the
kubelet fails to resolve reference "registry.
Severity
--------
Major
Steps to Reproduce
------------------
1)Upload vault and apply
system application-upload /usr/local/ share/applicati ons/helm/ vault-20. 06-14.tgz
system application-apply vault
2)Now monitor the pods status as it fails to download the image form local registry
[sysadmin@ controller- 0 ~(keystone_admin)]$ kubectl get pods -n vault agent-injector- db6878c69- v7ghr 0/1 ImagePullBackOff 0 10m controller- 0 ~(keystone_admin)]$
NAME READY STATUS RESTARTS AGE
sva-vault-0 0/1 ImagePullBackOff 0 10m
sva-vault-
sva-vault-manager-0 1/1 Running 0 10m
[sysadmin@
3)pod events
Events: eClaims hVolume 10m attachdetach- controller AttachVolume.Attach succeeded for volume "pvc-26224005- 38c7-4dc6- ae78-7d941867b1 61" hVolume 10m attachdetach- controller AttachVolume.Attach succeeded for volume "pvc-ede416a5- 6708-4d52- b4f6-3e95f37d79 a3" local:9001/ docker. io/vault: 1.4.2" local:9001/ docker. io/vault: 1.4.2": rpc error: code = Unknown desc = failed to pull and unpack image "registry. local:9001/ docker. io/vault: 1.4.2": failed to resolve reference "registry. local:9001/ docker. io/vault: 1.4.2": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed local:9001/ docker. io/vault: 1.4.2"
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 11m (x4 over 11m) default-scheduler running "VolumeBinding" filter plugin for pod "sva-vault-0": pod has unbound immediate PersistentVolum
Normal Scheduled 10m default-scheduler Successfully assigned vault/sva-vault-0 to controller-0
Normal SuccessfulAttac
Normal SuccessfulAttac
Normal Pulling 8m56s (x4 over 10m) kubelet, controller-0 Pulling image "registry.
Warning Failed 8m55s (x4 over 10m) kubelet, controller-0 Failed to pull image "registry.
Warning Failed 8m55s (x4 over 10m) kubelet, controller-0 Error: ErrImagePull
Warning Failed 8m29s (x7 over 10m) kubelet, controller-0 Error: ImagePullBackOff
Normal BackOff 22s (x42 over 10m) kubelet, controller-0 Back-off pulling image "registry.
4)Same issue for portieris images, i think it happens for any image from the local registry.
5)Also manual pull fails with "crictl"
[sysadmin@ controller- 0 ~(keystone_admin)]$ crictl pull registry. local:9001/ docker. io/vault: 1.4.2 local:9001/ docker. io/vault: 1.4.2": failed to resolve reference "registry. local:9001/ docker. io/vault: 1.4.2": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
FATA[0000] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry.
but works with docker pull
[sysadmin@ controller- 0 ~(keystone_admin)]$ sudo docker pull registry. local:9001/ docker. io/vault: 1.4.2 ======= =====> ] 17.83MB/49.32MB
Password:
1.4.2: Pulling from vault
21c83c524219: Pull complete
d710f8cd918f: Pull complete
fd8d5779418b: Extracting [======
a42e38f40189: Download complete
95183e98f2b7: Download complete
Expected Behavior
------------------
Images should be successfully downloaded from local registry
Actual Behavior
----------------
Images fail to download from the local registry
Reproducibility
---------------
Reproducible
System Configuration ------- ------
-------
standard controller duplex
Branch/Pull Time/Commit ------- ------- --
-------
master approximately 2021-01-13
Last Pass /review. opendev. org/c/starlingx /stx-puppet/ +/756559
---------
probably before:
https:/
Timestamp/Logs
--------------
n/a
Test Activity
-------------
manual verification of issue