STX-O Master failing to apply : "validate.nginx.ingress.kubernetes.io" denied the request

[Gabriel Calixto de Paula]

Brief Description
------------------

STX-Openstack master is failing to apply in STX master at 0%.
Severity
---------

Major

Steps to Reproduce
-------------------

    Install lab with STX master
    apply STX-O Master

Expected Behavior
------------------

STX-O master is applied

Actual Behavior
------------------

STX-O master fails to apply

Reproducibility
-----------------

Seen once

System Configuration
--------------------
 DX

Load info
--------------------
BUILD_DATE="2023-11-06 07:00:59 +0000"

STX-O:
https://mirror.starlingx.cengn.ca/mirror/starlingx/master/debian/openstack/20231101T160059Z/outputs/

Last Pass
---------------------

Aug

Timestamp/Logs
-----------------

Provide a snippet of logs if available and the timestamp when issue was seen.

Sysinv snippet:

sysinv 2023-11-07 03:06:37.223 21110 ERROR sysinv.conductor.kube_app [-] Application stx-openstack: release ingress: Failed during apply :Helm install failed: 1 error occurred: * admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: nginx.ingress.kubernetes.io/configuration-snippet annotation cannot be used. Snippet directives are disabled by the Ingress administrator

Last Helm logs:
: None: None2023-11-07 03:06:37.223 21110 ERROR sysinv.conductor.kube_app NoneType: None2023-11-07 03:06:37.223 21110 ERROR sysinv.conductor.kube_app

Alarms
---------------

[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+---------------------------+-------------------------------+----------+---------------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+---------------------------+-------------------------------+----------+---------------------+
| 750.002 | Application Apply Failure | k8s_application=stx-openstack | major | 2023-11-07T03:06:37 |
| | | | | .332281 |
| | | | | |
+----------+---------------------------+-------------------------------+----------+---------------------+

Test Activity
---------------

Sanity

Workaround
--------------
N/A

[Ghada Khalil (gkhalil)]

There was a recent upversion of nginx in the main branch:
https://review.opendev.org/c/starlingx/nginx-ingress-controller-armada-app/+/899410
https://review.opendev.org/c/starlingx/ansible-playbooks/+/899271

This could be the cause of the issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/nginx-ingress-controller-armada-app/+/901045

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/nginx-ingress-controller-armada-app/+/901045
Committed: https://opendev.org/starlingx/nginx-ingress-controller-armada-app/commit/556c6a09e21696843c78f72ae63069999bd46f67
Submitter: "Zuul (22348)"
Branch: master

commit 556c6a09e21696843c78f72ae63069999bd46f67
Author: amantri <email address hidden>
Date: Wed Nov 15 09:55:44 2023 -0500

    Enable "allow-snippet-annotations" in ingress-nginx static values

    nginx v1.9.0 onwards, "allow-snippet-annotations" is disabled
    by default due to security vulnerability reported here
    https://github.com/kubernetes/ingress-nginx/issues/7837,
    openstack failed to apply due to this change since it is using "configuration-snippet" under annotations in its openstack ingress definition.we are changing this default behavior to let openstack apply
    successfully until this upstream PR
    https://github.com/kubernetes/ingress-nginx/pull/9742 is addressed.
    once we upversion the nginx with the fix, we disable
    "allow-snippet-annotations" and openstack team will have to change
    their configuration.

    Test Cases:
    PASS: Enable "allow-snippet-annotations" in nginx configmap
          and apply the openstack app successfully
    PASS: Test stx-openstack with installation and verify openstack is
          applied successfully

    Closes-bug: 2042957

    Change-Id: Ic6c379803f17998ef7f573fa1fffa566b9e74e39
    Signed-off-by: amantri <email address hidden>