StarlingX

  1. Bug #1912696
  2. Activity log

Activity log for bug #1912696

Date Who What changed Old value New value Message
2021-01-21 21:43:53 Michel Thebeau [WIND] bug added bug
2021-01-21 23:39:24 Michel Thebeau [WIND] starlingx: assignee Michel Thebeau [WIND] (mthebeau)
2021-01-21 23:39:45 Michel Thebeau [WIND] starlingx: status New In Progress
2021-01-21 23:39:45 Michel Thebeau [WIND] starlingx: assignee Michel Thebeau [WIND] (mthebeau)
2021-01-21 23:40:27 Michel Thebeau [WIND] description Brief Description ----------------- Apply any application vault,portieris , the pods fails with "ImagePullBackOff" error since the kubelet fails to resolve reference "registry.local:9001/docker.io/hashicorp/vault-k8s:0.4.0"" Severity -------- Major Steps to Reproduce ------------------ 1)Upload vault and apply system application-upload /usr/local/share/applications/helm/vault-20.06-14.tgz system application-apply vault 2)Now monitor the pods status as it fails to download the image form local registry [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pods -n vault NAME READY STATUS RESTARTS AGE sva-vault-0 0/1 ImagePullBackOff 0 10m sva-vault-agent-injector-db6878c69-v7ghr 0/1 ImagePullBackOff 0 10m sva-vault-manager-0 1/1 Running 0 10m [sysadmin@controller-0 ~(keystone_admin)]$ 3)pod events Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 11m (x4 over 11m) default-scheduler running "VolumeBinding" filter plugin for pod "sva-vault-0": pod has unbound immediate PersistentVolumeClaims Normal Scheduled 10m default-scheduler Successfully assigned vault/sva-vault-0 to controller-0 Normal SuccessfulAttachVolume 10m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-26224005-38c7-4dc6-ae78-7d941867b161" Normal SuccessfulAttachVolume 10m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-ede416a5-6708-4d52-b4f6-3e95f37d79a3" Normal Pulling 8m56s (x4 over 10m) kubelet, controller-0 Pulling image "registry.local:9001/docker.io/vault:1.4.2" Warning Failed 8m55s (x4 over 10m) kubelet, controller-0 Failed to pull image "registry.local:9001/docker.io/vault:1.4.2": rpc error: code = Unknown desc = failed to pull and unpack image "registry.local:9001/docker.io/vault:1.4.2": failed to resolve reference "registry.local:9001/docker.io/vault:1.4.2": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed Warning Failed 8m55s (x4 over 10m) kubelet, controller-0 Error: ErrImagePull Warning Failed 8m29s (x7 over 10m) kubelet, controller-0 Error: ImagePullBackOff Normal BackOff 22s (x42 over 10m) kubelet, controller-0 Back-off pulling image "registry.local:9001/docker.io/vault:1.4.2" 4)Same issue for portieris images, i think it happens for any image from the local registry. 5)Also manual pull fails with "crictl" [sysadmin@controller-0 ~(keystone_admin)]$ crictl pull registry.local:9001/docker.io/vault:1.4.2 FATA[0000] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry.local:9001/docker.io/vault:1.4.2": failed to resolve reference "registry.local:9001/docker.io/vault:1.4.2": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed but works with docker pull [sysadmin@controller-0 ~(keystone_admin)]$ sudo docker pull registry.local:9001/docker.io/vault:1.4.2 Password: 1.4.2: Pulling from vault 21c83c524219: Pull complete d710f8cd918f: Pull complete fd8d5779418b: Extracting [==================> ] 17.83MB/49.32MB a42e38f40189: Download complete 95183e98f2b7: Download complete Expected Behavior ------------------ Images should be successfully downloaded from local registry Actual Behavior ---------------- Images fail to download from the local registry Reproducibility --------------- Reproducible System Configuration -------------------- standard controller duplex Branch/Pull Time/Commit ----------------------- master approximately 2021-01-13 Last Pass --------- probably before: https://review.opendev.org/c/starlingx/stx-puppet/+/756559 Timestamp/Logs -------------- n/a Test Activity ------------- manual verification of issue Brief Description ----------------- Apply any application vault,portieris , the pods fails with "ImagePullBackOff" error since the kubelet fails to resolve reference "registry.local:9001/docker.io/hashicorp/vault-k8s:0.4.0"" Severity -------- Major Steps to Reproduce ------------------ 1)Upload vault and apply system application-upload /usr/local/share/applications/helm/vault-*.tgz system application-apply vault 2)Now monitor the pods status as it fails to download the image form local registry [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pods -n vault NAME READY STATUS RESTARTS AGE sva-vault-0 0/1 ImagePullBackOff 0 10m sva-vault-agent-injector-db6878c69-v7ghr 0/1 ImagePullBackOff 0 10m sva-vault-manager-0 1/1 Running 0 10m [sysadmin@controller-0 ~(keystone_admin)]$ 3)pod events Events:   Type Reason Age From Message   ---- ------ ---- ---- -------   Warning FailedScheduling 11m (x4 over 11m) default-scheduler running "VolumeBinding" filter plugin for pod "sva-vault-0": pod has unbound immediate PersistentVolumeClaims   Normal Scheduled 10m default-scheduler Successfully assigned vault/sva-vault-0 to controller-0   Normal SuccessfulAttachVolume 10m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-26224005-38c7-4dc6-ae78-7d941867b161"   Normal SuccessfulAttachVolume 10m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-ede416a5-6708-4d52-b4f6-3e95f37d79a3"   Normal Pulling 8m56s (x4 over 10m) kubelet, controller-0 Pulling image "registry.local:9001/docker.io/vault:1.4.2"   Warning Failed 8m55s (x4 over 10m) kubelet, controller-0 Failed to pull image "registry.local:9001/docker.io/vault:1.4.2": rpc error: code = Unknown desc = failed to pull and unpack image "registry.local:9001/docker.io/vault:1.4.2": failed to resolve reference "registry.local:9001/docker.io/vault:1.4.2": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed   Warning Failed 8m55s (x4 over 10m) kubelet, controller-0 Error: ErrImagePull   Warning Failed 8m29s (x7 over 10m) kubelet, controller-0 Error: ImagePullBackOff   Normal BackOff 22s (x42 over 10m) kubelet, controller-0 Back-off pulling image "registry.local:9001/docker.io/vault:1.4.2" 4)Same issue for portieris images, i think it happens for any image from the local registry. 5)Also manual pull fails with "crictl" [sysadmin@controller-0 ~(keystone_admin)]$ crictl pull registry.local:9001/docker.io/vault:1.4.2 FATA[0000] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry.local:9001/docker.io/vault:1.4.2": failed to resolve reference "registry.local:9001/docker.io/vault:1.4.2": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed but works with docker pull [sysadmin@controller-0 ~(keystone_admin)]$ sudo docker pull registry.local:9001/docker.io/vault:1.4.2 Password: 1.4.2: Pulling from vault 21c83c524219: Pull complete d710f8cd918f: Pull complete fd8d5779418b: Extracting [==================> ] 17.83MB/49.32MB a42e38f40189: Download complete 95183e98f2b7: Download complete Expected Behavior ------------------ Images should be successfully downloaded from local registry Actual Behavior ---------------- Images fail to download from the local registry Reproducibility --------------- Reproducible System Configuration -------------------- standard controller duplex Branch/Pull Time/Commit ----------------------- master approximately 2021-01-13 Last Pass --------- probably before:  https://review.opendev.org/c/starlingx/stx-puppet/+/756559 Timestamp/Logs -------------- n/a Test Activity ------------- manual verification of issue
2021-01-22 13:32:29 Ghada Khalil starlingx: assignee Michel Thebeau [WIND] (mthebeau)
2021-01-22 13:32:34 Ghada Khalil starlingx: importance Undecided High
2021-01-22 13:32:52 Ghada Khalil tags stx.5.0 stx.apps stx.containers
2021-01-27 13:38:45 Michel Thebeau [WIND] starlingx: status In Progress Fix Committed
2021-01-27 16:02:08 Ghada Khalil starlingx: status Fix Committed Fix Released
2021-03-24 15:06:00 Michel Thebeau [WIND] starlingx: status Fix Released In Progress
2021-03-29 15:38:56 Michel Thebeau [WIND] starlingx: status In Progress Fix Released