Cannot obtain armada lock when using k8s 1.23

[Dan Voiculeasa]

Brief Description
StarlingX now uses k8s 1.23.1 by default: https://review.opendev.org/c/starlingx/tools/+/837085
Armada needs to be updated to support the APIs. APIs were deprecated in v1.22.

Severity
-----
major

Steps to Reproduce
----
system application apply an app

Expected Behavior
---------------------
Apply application should not have failed

Actual Behavior
---------------
Application apply

Reproducibility
----------------
yes

System Configuration
--------
any

Load info (eg: 2022-03-10_20-00-07)
-------------------
Build ID: 2022-06-07_19-00-50

Last Pass
--------------
Before https://review.opendev.org/c/starlingx/tools/+/837085 merged.

Timestamp/Logs
--------------

2022-06-08 16:13:25.699 109 DEBUG armada.handlers.document [-] Resolving reference /tmp/manifests/hello-kitty/1.16/hello-kitty-manifest.yaml. resolve_reference /usr/local/lib/python3.6/dist-packages/armada/handlers/document.py:51
2022-06-08 16:13:25.701 109 DEBUG armada.handlers.tiller [-] Using Tiller host IP: 127.0.0.1 _get_tiller_ip /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:164
2022-06-08 16:13:25.701 109 DEBUG armada.handlers.tiller [-] Using Tiller host port: 24134 _get_tiller_port /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:173
2022-06-08 16:13:25.702 109 DEBUG armada.handlers.tiller [-] Tiller getting gRPC insecure channel at 127.0.0.1:24134 with options: [grpc.max_send_message_length=429496729, grpc.max_receive_message_length=429496729] get_channel /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:126
2022-06-08 16:13:25.711 109 DEBUG armada.handlers.tiller [-] Armada is using Tiller at: 127.0.0.1:24134, namespace=kube-system, timeout=300 __init__ /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:106
2022-06-08 16:13:25.712 109 INFO armada.handlers.lock [-] Acquiring lock
2022-06-08 16:13:25.721 109 INFO armada.handlers.lock [-] Lock Custom Resource Definition not found, creating now
2022-06-08 16:13:25.727 109 ERROR armada.cli [-] Caught unexpected exception: kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '5af7ff29-3ca7-49b5-8327-bfdad63e5a53', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '0a21969f-da87-4311-9dac-cde1da033e56', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'ea173ebd-1b7c-4880-a201-451bac304962', 'Date': 'Wed, 08 Jun 2022 16:13:25 GMT', 'Content-Length': '174'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server could not find the requested resource","reason":"NotFound","details":{},"code":404}
2022-06-08 16:13:25.727 109 ERROR armada.cli Traceback (most recent call last):
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 137, in acquire_lock
2022-06-08 16:13:25.727 109 ERROR armada.cli self.lock_config.create_lock()
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 216, in create_lock
2022-06-08 16:13:25.727 109 ERROR armada.cli body=self.body)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/k8s.py", line 382, in create_custom_resource
2022-06-08 16:13:25.727 109 ERROR armada.cli group, version, namespace, plural, body)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api/custom_objects_api.py", line 225, in create_namespaced_custom_object
2022-06-08 16:13:25.727 109 ERROR armada.cli return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs) # noqa: E501
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api/custom_objects_api.py", line 358, in create_namespaced_custom_object_with_http_info
2022-06-08 16:13:25.727 109 ERROR armada.cli collection_formats=collection_formats)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 353, in call_api
2022-06-08 16:13:25.727 109 ERROR armada.cli _preload_content, _request_timeout, _host)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 184, in __call_api
2022-06-08 16:13:25.727 109 ERROR armada.cli _request_timeout=_request_timeout)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 397, in request
2022-06-08 16:13:25.727 109 ERROR armada.cli body=body)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 281, in POST
2022-06-08 16:13:25.727 109 ERROR armada.cli body=body)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 234, in request
2022-06-08 16:13:25.727 109 ERROR armada.cli raise ApiException(http_resp=r)
2022-06-08 16:13:25.727 109 ERROR armada.cli kubernetes.client.exceptions.ApiException: (404)
2022-06-08 16:13:25.727 109 ERROR armada.cli Reason: Not Found
2022-06-08 16:13:25.727 109 ERROR armada.cli HTTP response headers: HTTPHeaderDict({'Audit-Id': '59f812c5-b579-4f35-9845-b76b6fda04eb', 'Cache-Control': 'no-cache, private', 'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '0a21969f-da87-4311-9dac-cde1da033e56', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'ea173ebd-1b7c-4880-a201-451bac304962', 'Date': 'Wed, 08 Jun 2022 16:13:25 GMT', 'Content-Length': '19'})
2022-06-08 16:13:25.727 109 ERROR armada.cli HTTP response body: 404 page not found
2022-06-08 16:13:25.727 109 ERROR armada.cli
2022-06-08 16:13:25.727 109 ERROR armada.cli
2022-06-08 16:13:25.727 109 ERROR armada.cli
2022-06-08 16:13:25.727 109 ERROR armada.cli During handling of the above exception, another exception occurred:
2022-06-08 16:13:25.727 109 ERROR armada.cli
2022-06-08 16:13:25.727 109 ERROR armada.cli Traceback (most recent call last):
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2022-06-08 16:13:25.727 109 ERROR armada.cli self.invoke()
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 219, in invoke
2022-06-08 16:13:25.727 109 ERROR armada.cli resp = self.handle(documents, tiller)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 72, in func_wrapper
2022-06-08 16:13:25.727 109 ERROR armada.cli with Lock(lock_name, bearer_token=bearer_token) as lock:
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 180, in __enter__
2022-06-08 16:13:25.727 109 ERROR armada.cli self.acquire_lock()
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 144, in acquire_lock
2022-06-08 16:13:25.727 109 ERROR armada.cli self.lock_config.create_definition()
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 304, in create_definition
2022-06-08 16:13:25.727 109 ERROR armada.cli self.k8s.create_custom_resource_definition(crd)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/k8s.py", line 367, in create_custom_resource_definition
2022-06-08 16:13:25.727 109 ERROR armada.cli return self.api_extensions.create_custom_resource_definition(crd)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api/apiextensions_v1beta1_api.py", line 65, in create_custom_resource_definition
2022-06-08 16:13:25.727 109 ERROR armada.cli return self.create_custom_resource_definition_with_http_info(body, **kwargs) # noqa: E501
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api/apiextensions_v1beta1_api.py", line 166, in create_custom_resource_definition_with_http_info
2022-06-08 16:13:25.727 109 ERROR armada.cli collection_formats=collection_formats)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 353, in call_api
2022-06-08 16:13:25.727 109 ERROR armada.cli _preload_content, _request_timeout, _host)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 184, in __call_api
2022-06-08 16:13:25.727 109 ERROR armada.cli _request_timeout=_request_timeout)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 397, in request
2022-06-08 16:13:25.727 109 ERROR armada.cli body=body)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 281, in POST
2022-06-08 16:13:25.727 109 ERROR armada.cli body=body)
2022-06-08 16:13:25.727 109 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 234, in request
2022-06-08 16:13:25.727 109 ERROR armada.cli raise ApiException(http_resp=r)
2022-06-08 16:13:25.727 109 ERROR armada.cli kubernetes.client.exceptions.ApiException: (404)
2022-06-08 16:13:25.727 109 ERROR armada.cli Reason: Not Found
2022-06-08 16:13:25.727 109 ERROR armada.cli HTTP response headers: HTTPHeaderDict({'Audit-Id': '5af7ff29-3ca7-49b5-8327-bfdad63e5a53', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '0a21969f-da87-4311-9dac-cde1da033e56', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'ea173ebd-1b7c-4880-a201-451bac304962', 'Date': 'Wed, 08 Jun 2022 16:13:25 GMT', 'Content-Length': '174'})
2022-06-08 16:13:25.727 109 ERROR armada.cli HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server could not find the requested resource","reason":"NotFound","details":{},"code":404}
2022-06-08 16:13:25.727 109 ERROR armada.cli
2022-06-08 16:13:25.727 109 ERROR armada.cli
2022-06-08 16:13:25.727 109 ERROR armada.cli
command terminated with exit code 1

Test Activity
------------
N/A

Workaround
-------------
N/A

[Dan Voiculeasa (dvoicule)]

Identified upstream commit that fixes the issue: https://opendev.org/airship/armada/commit/c5d39f27cacaa953be43c7d3265bb693db0939d0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/integ/+/845895

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/integ/+/845752
Committed: https://opendev.org/starlingx/integ/commit/0783ec950b424c980658765515a649c846dc5121
Submitter: "Zuul (22348)"
Branch: master

commit 0783ec950b424c980658765515a649c846dc5121
Author: Dan Voiculeasa <email address hidden>
Date: Tue Jun 14 16:10:32 2022 +0300

    Debian: Upversion armada sources for container build

    Currently, we use in the playbooks [1] an armada image built upstream
    [2]. We use armada from upstream helm2 branch.

    Armada needs to be patched to add support for k8s >=1.22.
    Proposed an upstream patch, but we don't know how long it takes until
    it is merged:
    https://review.opendev.org/c/airship/armada/+/845392

    Instead of waiting for upstream commit to merge, and an image be
    generated, we provide the code change here, so an armada image with
    k8s >=1.22 support can be generated. The k8s >=1.22 support is added
    by patch 0003.
    Necessary StarlingX build changes to support generating an container
    image and push to https://hub.docker.com/r/starlingx will be
    addressed in another commit.

    Note: since we always used an upstream built armada container, this
    package purpose was to provide helm chart overrides. We add a new
    purpose: to release to opensource the changes we are about to do to
    armada, since we'll be building a container image using these changes.

    To achieve this we do the following:
    - upversion sources from 7ef4b8643b5ec5216a8f6726841e156c0aa54a1a
    to ddbdd7256c20f138737f6cbd772312f7a19f58b8. This ensures we are
    patching the image used in the playbooks[2].
    - create patches 0001 and 0002 to ensure there are no helm chart
    changes between upversion. This reduces testing effort related to
    original purpose (provide helm chart overrides) of this package.
    - create patch 0003 to add k8s >=1.22 support.
    - old patches are not changed, but renamed from 0001->0005 to
    0004->0008 and regenerated.

    Other notes:
    We don't need to port this work to CentOS. This work is supposed to be
    temporary until the upstream airship/armada commit merges.

    Tests on Debian:
    PASS: build-pkgs -c -p armada
    PASS: make images
          Upload image to controller, use it.
          Using the new armada image do an apply,remove,apply,remove,apply
          chain for a custom StarlingX app.

    [1]: https://opendev.org/starlingx/ansible-playbooks/src/branch/
    master/playbookconfig/src/playbooks/roles/common/
    load-images-information/vars/k8s-v1.22.5/system-images.yml#L5
    [2] quay.io/airshipit/armada:
    ddbdd7256c20f138737f6cbd772312f7a19f58b8-ubuntu_bionic

    Partial-Bug: 1978409
    Signed-off-by: Dan Voiculeasa <email address hidden>
    Change-Id: Id51c241a3965ef462d325da4ffce37a81693a9f4

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/root/+/846020

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/846024

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/integ/+/845895
Committed: https://opendev.org/starlingx/integ/commit/537bffe16ea2d4188589a4d707e9364850177467
Submitter: "Zuul (22348)"
Branch: master

commit 537bffe16ea2d4188589a4d707e9364850177467
Author: Dan Voiculeasa <email address hidden>
Date: Tue Jun 14 23:23:08 2022 +0300

    Add support for building armada image

    Follow-up to https://review.opendev.org/c/starlingx/integ/+/845752.
    As explained in the mentioned commit we need to build the armada
    container image using the sources provided there.
    Add support for that.

    Partial-Bug: 1978409
    Signed-off-by: Dan Voiculeasa <email address hidden>
    Change-Id: I26e8117272a6925ea429be10be91ce20f6b71c8b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/846020
Committed: https://opendev.org/starlingx/root/commit/f4e83941a2362619796c09aca2b15c300f2af87f
Submitter: "Zuul (22348)"
Branch: master

commit f4e83941a2362619796c09aca2b15c300f2af87f
Author: Dan Voiculeasa <email address hidden>
Date: Wed Jun 15 18:52:07 2022 +0300

    Add armada-image container tag

    Partial-Bug: 1978409
    Signed-off-by: Dan Voiculeasa <email address hidden>
    Change-Id: Id6854aab3933d8f1e87cd1435aa6c5cb73dddca1

[Ghada Khalil (gkhalil)]

Marking as Fix Released based on above fixes. If further follow-up is required, a new LP should be opened.

[Ghada Khalil (gkhalil)]

Re-opening as it appears there is still one starlingx review that will need to merge before this can be considered complete: https://review.opendev.org/c/starlingx/ansible-playbooks/+/846024

[Ghada Khalil (gkhalil)]

Changing the priority to high as this fix is required to enable stx-openstack to work with armada and k8s 1.23

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/846024
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/cca493d8de5a232eb592ba64d6f852b775034bf2
Submitter: "Zuul (22348)"
Branch: master

commit cca493d8de5a232eb592ba64d6f852b775034bf2
Author: Dan Voiculeasa <email address hidden>
Date: Wed Jun 15 18:45:31 2022 +0300

    Update armada image for k8s >=1.22

    We observe that CRD was changed [1] in k8s 1.22.
    Need to update armada container due to API changes.
    This work will only enable StarlingX armada apps when doing a new
    deployment. For Upgrades path there will be another commit if
    required.

    Use a custom StarlingX updated armada image that can handle the
    API changes for k8s >= 1.22.

    Tests on AIO-SX Debian:
    PASS: bootstrap
    PASS: unlock
    PASS: install StarlingX armada app

    [1]: https://kubernetes.io/docs/reference/using-api/
    deprecation-guide/#customresourcedefinition-v122

    Depends-On: https://review.opendev.org/c/starlingx/integ/+/845752
    Depends-On: https://review.opendev.org/c/starlingx/integ/+/845895
    Depends-On: https://review.opendev.org/c/starlingx/root/+/846020
    Closes-Bug: 1978409
    Signed-off-by: Dan Voiculeasa <email address hidden>
    Change-Id: I273b958b5dc24f7776601d3423a41acf3bfe7b6c