nginx-ingress-controller apply-fail

[Alexandru Dimofte]

Brief Description
-----------------
nginx-ingress-controller apply-fail during stx setup and this blocks Daily Sanity

Severity
--------
<Critical: System/Feature is not usable due to the defect>

Steps to Reproduce
------------------
Try to install stx image 20210319T013254Z

Expected Behavior
------------------
Image installation should work fine.

Actual Behavior
----------------
StarlingX install fail at the Setup step because nginx-ingress-controller apply fails.

Reproducibility
---------------
100%

System Configuration
--------------------
One node system, Two node system, Multi-node system, Dedicated storage

Branch/Pull Time/Commit
-----------------------
master

Last Pass
---------
20210318T063229Z

Timestamp/Logs
--------------
will be attached

Test Activity
-------------
Sanity

Workaround
----------
-

[Alexandru Dimofte (adimofte)]

2021-03-19 10:19:18,116 p=13702 u=sysadmin | TASK [bootstrap/bringup-bootstrap-applications : Wait until application is in the applied state] ***
2021-03-19 10:19:20,243 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (30 retries left).
2021-03-19 10:19:42,398 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (29 retries left).
2021-03-19 10:20:04,106 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (28 retries left).
2021-03-19 10:20:25,803 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (27 retries left).
2021-03-19 10:20:47,504 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (26 retries left).
2021-03-19 10:21:09,200 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (25 retries left).
2021-03-19 10:21:30,871 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (24 retries left).
2021-03-19 10:21:52,577 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (23 retries left).
2021-03-19 10:22:14,303 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (22 retries left).
2021-03-19 10:22:36,020 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (21 retries left).
2021-03-19 10:22:57,754 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (20 retries left).
2021-03-19 10:23:19,471 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (19 retries left).
2021-03-19 10:23:41,158 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (18 retries left).
2021-03-19 10:24:03,004 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (17 retries left).
2021-03-19 10:24:24,686 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (16 retries left).
2021-03-19 10:24:46,431 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (15 retries left).
2021-03-19 10:25:08,182 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (14 retries left).
2021-03-19 10:25:29,847 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (13 retries left).
2021-03-19 10:25:51,560 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (12 retries left).
2021-03-19 10:26:13,285 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (11 retries left).
2021-03-19 10:26:34,982 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (10 retries left).
2021-03-19 10:26:56,701 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (9 retries left).
2021-03-19 10:27:18,450 p=13702 u=sysadmin | FAILED - RETRYING: Wait until application is in the applied state (8 retries left).
2021-03-19 10:27:40,109 p=13702 u=sysadmin...

[Alexandru Dimofte (adimofte)]

I suspect:
./cgcs-root/stx/nginx-ingress-controller-armada-app 437db6f68af305a1667a5ac5ec722d96a1c44c9c 2021-03-17 12:28:58 -0400 Iago Regiani <email address hidden> Updated nginx-ingress controller to 0.41.2

[Ghada Khalil (gkhalil)]

Yes the issue is introduced by the nginx upversion. The upversion was reverted for now:
https://review.opendev.org/c/starlingx/nginx-ingress-controller-armada-app/+/782122

so this should be resolved in tonight's build.

The real resolution is for anyone using a private registry to populate their registry with the new nginx image. The typical email sent to the starlingx-discuss list was missed. An email will be sent before the code is re-merged.

[Alexandru Dimofte (adimofte)]

The issue is no more visible today using this image: 20210322T230300Z. I guess the bug can be closed now...

[Ghada Khalil (gkhalil)]

Email sent regarding the new container images required to populate the private registries:
http://lists.starlingx.io/pipermail/starlingx-discuss/2021-March/011044.html