Applying metrics-server failed to enable metrics server sample app

[Luiz Felipe Kina]

Brief Description
-----------------
On the 22.12 Duplex system, after enabled metric sample app for application 'metrics-server', applying it failed and the pod 'metric-server-test-app-deployment' can't start up due to failure to pull container image.

Severity
--------
Minor

Steps to Reproduce
------------------
Upload metrics-server
Apply metrics-server
Helm override update with sampleApp.create=true for metrics-server helm
Apply metrics-server

Expected Behavior
------------------
Metrics-server is applied and sample-app pod is running

Actual Behavior
----------------
Metric-server-test-pod is not working

Reproducibility
---------------
100% Reproducible.

System Configuration
--------------------
AIO-DX

Branch/Pull Time/Commit
-----------------------
Master

Last Pass
---------
N/a

Timestamp/Logs
--------------
  At about 2023-08-28 20:16, enabled metric sample app for application 'metrics-server', then applied it.

2023-08-28T20:16:42.637 controller-0 -bash: info HISTORY: PID=1263423 UID=42425 system helm-override-update --reuse-values --set sampleApp.create=true metrics-server metrics-server metrics-server
2023-08-28T20:17:05.159 controller-0 -bash: info HISTORY: PID=1263423 UID=42425 system application-apply metrics-server

  The application apply completed, but the pod 'metric-server-test-app-deployment' can't start up due to failure to pull container image 'stx-metrics-server-sample-app'.

sysinv 2023-08-28 20:17:14.194 2957921 INFO sysinv.conductor.kube_app [-] Application metrics-server (22.12-1) apply completed.

2023-08-28T20:18:07Z metric-server-test-app-deployment-6fd4b5c5df-rvh2c Pod Back-off pulling image "registry.local:9001/docker.io/starlingx/stx-metrics-server-sample-app:stx.7.0-v1.0.0" BackOff Normal

  Then tried to re-apply it, but it failed.

2023-08-28T20:18:50.818 controller-0 -bash: info HISTORY: PID=1263423 UID=42425 system helm-override-update --reuse-values --set sampleApp.create=true metrics-server metrics-server metrics-server
2023-08-28T20:19:00.637 controller-0 -bash: info HISTORY: PID=1263423 UID=42425 system application-apply metrics-server

sysinv 2023-08-28 20:48:04.996 2957921 ERROR sysinv.conductor.kube_app [-] Application metrics-server: release metrics-server: Failed during apply :Helm upgrade failed: timed out waiting for the condition
waiting for release ms-metrics-server resources (created: 5 updated: 9 deleted: 0)

  sysinv just downloaded the container image 'stx-metrics-server-sample-app' on controller-0, but the pod 'metric-server-test-app-deployment' was created on controller-1, this is why the pod failed to pull the container image.

sysadmin@controller-0:~$ crictl images | grep stx-metrics-server-sample-app
registry.local:9001/docker.io/starlingx/stx-metrics-server-sample-app stx.7.0-v1.0.0 8772b33471f50 41.2MB

sysadmin@controller-1:~$ crictl image | grep stx-metrics-server-sample-app
sysadmin@controller-1:~$

Test Activity
-------------
Feature testing

[Felipe Sanches Zanoni (fsanches)]

Review: https://review.opendev.org/c/starlingx/metrics-server-armada-app/+/896234

[Ghada Khalil (gkhalil)]

The associated review is merged. The correct state is Fix Released.