DX installation failed to pull images

[Peng Peng]

Brief Description
-----------------
STX DX installation failed. The puppet log show

/Stage[main]/Platform::Kubernetes::Master::Init/Platform::Kubernetes::Pull_images_from_registry[pull images from private registry]/Exec[pre pull k8s images]/returns: time="2023-10-10T18:46:52Z" level=fatal msg="pulling image: rpc error: code = DeadlineExceeded desc = failed to pull and unpack image \"registry.local:9001/registry.k8s.io/kube-apiserver:v1.25.3\": failed to resolve reference \"registry.local:9001/registry.k8s.io/kube-apiserver:v1.25.3\": failed to authorize: failed to fetch oauth token: Post \"https://128.224.151.54:9002/token/\": dial tcp 128.224.151.54:9002: i/o timeout"^[[0m
t

Severity
--------
Major

Steps to Reproduce
------------------
install DX

TC-name:

Expected Behavior
------------------
DX installation success

Actual Behavior
----------------
DX installation failed

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Two node system

Lab-name: SM_5-6

Branch/Pull Time/Commit
-----------------------
BUILD_ID="20231010T060059Z"
JOB="STX_build_debian_master"

Last Pass
---------
https://mirror.starlingx.cengn.ca/mirror/starlingx/master/debian/monolithic/20231002T060059Z/outputs/iso/starlingx-intel-x86-64-cd.iso

Timestamp/Logs
--------------
[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get hosts -n=deployment -o=wide
NAME ADMINISTRATIVE OPERATIONAL AVAILABILITY PROFILE INSYNC SCOPE RECONCILED
controller-0 unlocked enabled available controller-0-profile true bootstrap true
controller-1 unlocked disabled offline controller-0-profile false bootstrap false

2023-10-10T18:49:53.050 ^[[mNotice: 2023-10-10 18:49:53 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Platform::Kubernetes::Pull_images_from_registry[pull images from private registry]/Exec[pre pull k8s images]/returns: time="2023-10-10T18:46:52Z" level=fatal msg="pulling image: rpc error: code = DeadlineExceeded desc = failed to pull and unpack image \"registry.local:9001/registry.k8s.io/kube-apiserver:v1.25.3\": failed to resolve reference \"registry.local:9001/registry.k8s.io/kube-apiserver:v1.25.3\": failed to authorize: failed to fetch oauth token: Post \"https://128.224.151.54:9002/token/\": dial tcp 128.224.151.54:9002: i/o timeout"^[[0m

Automation log:
http://128.224.150.21/auto_logs/sys_install/stx/sm_5_6/202310101304

Test Activity
-------------
Installation

[Ghada Khalil (gkhalil)]

Assigning to Andre.

I believe this issue was introduced by the following recent code change: https://review.opendev.org/c/starlingx/stx-puppet/+/897467 which merged on Oct 6. LP: https://bugs.launchpad.net/starlingx/+bug/2038550

A follow-up commit was merged on Oct 10: https://review.opendev.org/c/starlingx/stx-puppet/+/897873

Need Andre to confirm that this sanity issue is now addressed by the follow-up commit. Once confirmed, this LP can be marked as Fix Released.

[Andre Kantek (akantek)]

The https://review.opendev.org/c/starlingx/stx-puppet/+/897873 is merged:
https://opendev.org/starlingx/stx-puppet/commit/58581b88e9c21c990dd0219bd2278fc40b42ed8f