kubelet fails to resolve 'registry.local' while downloading vault/portieris images from local registry

[Michel Thebeau [WIND]]

Brief Description
-----------------
Apply any application vault,portieris , the pods fails with "ImagePullBackOff" error since the
kubelet fails to resolve reference "registry.local:9001/docker.io/hashicorp/vault-k8s:0.4.0""

Severity
--------
Major

Steps to Reproduce
------------------
1)Upload vault and apply

system application-upload /usr/local/share/applications/helm/vault-*.tgz
system application-apply vault
2)Now monitor the pods status as it fails to download the image form local registry

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
sva-vault-0 0/1 ImagePullBackOff 0 10m
sva-vault-agent-injector-db6878c69-v7ghr 0/1 ImagePullBackOff 0 10m
sva-vault-manager-0 1/1 Running 0 10m
[sysadmin@controller-0 ~(keystone_admin)]$
3)pod events

Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning FailedScheduling 11m (x4 over 11m) default-scheduler running "VolumeBinding" filter plugin for pod "sva-vault-0": pod has unbound immediate PersistentVolumeClaims
  Normal Scheduled 10m default-scheduler Successfully assigned vault/sva-vault-0 to controller-0
  Normal SuccessfulAttachVolume 10m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-26224005-38c7-4dc6-ae78-7d941867b161"
  Normal SuccessfulAttachVolume 10m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-ede416a5-6708-4d52-b4f6-3e95f37d79a3"
  Normal Pulling 8m56s (x4 over 10m) kubelet, controller-0 Pulling image "registry.local:9001/docker.io/vault:1.4.2"
  Warning Failed 8m55s (x4 over 10m) kubelet, controller-0 Failed to pull image "registry.local:9001/docker.io/vault:1.4.2": rpc error: code = Unknown desc = failed to pull and unpack image "registry.local:9001/docker.io/vault:1.4.2": failed to resolve reference "registry.local:9001/docker.io/vault:1.4.2": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning Failed 8m55s (x4 over 10m) kubelet, controller-0 Error: ErrImagePull
  Warning Failed 8m29s (x7 over 10m) kubelet, controller-0 Error: ImagePullBackOff
  Normal BackOff 22s (x42 over 10m) kubelet, controller-0 Back-off pulling image "registry.local:9001/docker.io/vault:1.4.2"
4)Same issue for portieris images, i think it happens for any image from the local registry.

5)Also manual pull fails with "crictl"

[sysadmin@controller-0 ~(keystone_admin)]$ crictl pull registry.local:9001/docker.io/vault:1.4.2
FATA[0000] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry.local:9001/docker.io/vault:1.4.2": failed to resolve reference "registry.local:9001/docker.io/vault:1.4.2": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

but works with docker pull

[sysadmin@controller-0 ~(keystone_admin)]$ sudo docker pull registry.local:9001/docker.io/vault:1.4.2
Password:
1.4.2: Pulling from vault
21c83c524219: Pull complete
d710f8cd918f: Pull complete
fd8d5779418b: Extracting [==================> ] 17.83MB/49.32MB
a42e38f40189: Download complete
95183e98f2b7: Download complete
Expected Behavior
------------------
Images should be successfully downloaded from local registry

Actual Behavior
----------------
Images fail to download from the local registry

Reproducibility
---------------
Reproducible

System Configuration
--------------------
standard controller duplex

Branch/Pull Time/Commit
-----------------------
master approximately 2021-01-13

Last Pass
---------
probably before:
 https://review.opendev.org/c/starlingx/stx-puppet/+/756559

Timestamp/Logs
--------------
n/a

Test Activity
-------------
manual verification of issue

[Ghada Khalil (gkhalil)]

stx.5.0 / high - this issue results in the vault and portieris applications being unusable

[Michel Thebeau [WIND] (mthebeau)]

Partial fix for this reported issue:

Review for portieris:
https://review.opendev.org/c/starlingx/portieris-armada-app/+/772449

And Merged for portieris:
commit edc1c00f5488197d8a3544be18f398675d189bec

https://review.opendev.org/plugins/gitiles/starlingx/portieris-armada-app/+/edc1c00f5488197d8a3544be18f398675d189bec

[Michel Thebeau [WIND] (mthebeau)]

Second fix for this issue:

Review for vault:
https://review.opendev.org/c/starlingx/vault-armada-app/+/772616

Merged for vault:
commit 53ad52c956314ebc00665f656ab2c4c4f49ff3e2

https://review.opendev.org/plugins/gitiles/starlingx/vault-armada-app/+/53ad52c956314ebc00665f656ab2c4c4f49ff3e2

[Michel Thebeau [WIND] (mthebeau)]

Revision for vault helm chart:

Review:
https://review.opendev.org/c/starlingx/vault-armada-app/+/782779

[Michel Thebeau [WIND] (mthebeau)]

Merged to master:

https://review.opendev.org/plugins/gitiles/starlingx/vault-armada-app/+/b85175197008df8bbff659524d577138f9d9ad99