Custom app apply-failed by Unable to acquire lock

[Peng Peng]

Brief Description
-----------------
Custom application failed to apply. log shows,
Unable to acquire lock before timeout

Severity
--------
Major

Steps to Reproduce
------------------
upload hello-kitty
apply hello-kitty
check application status

TC-name: z_containers/test_custom_containers.py::test_launch_app_via_sysinv

Expected Behavior
------------------
application apply successfully

Actual Behavior
----------------
application apply-failed

Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor

System Configuration
--------------------
Multi-node system

Lab-name: WP_8-12

Branch/Pull Time/Commit
-----------------------
2020-05-24_20-00-00

Last Pass
---------
2020-05-22_20-00-00

Timestamp/Logs
--------------
[2020-05-25 08:45:35,924] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[face::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-apply hello-kitty'
[2020-05-25 08:45:37,667] 436 DEBUG MainThread ssh.expect :: Output:
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | False |
| app_version | 1.16 |
| created_at | 2020-05-25T08:45:33.348435+00:00 |
| manifest_file | manifest.yaml |
| manifest_name | hello-kitty |
| name | hello-kitty |
| progress | None |
| status | applying |
| updated_at | 2020-05-25T08:45:35.464917+00:00 |
+---------------+----------------------------------+

[2020-05-25 08:47:00,481] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[face::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2020-05-25 08:47:01,859] 436 DEBUG MainThread ssh.expect :: Output:
+--------------------------+---------+-----------------------------------+----------------------------------------+--------------+------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+--------------------------+---------+-----------------------------------+----------------------------------------+--------------+------------------------------------------+
| cert-manager | 1.0-0 | cert-manager-manifest | certmanager-manifest.yaml | applied | completed |
| hello-kitty | 1.16 | hello-kitty | manifest.yaml | apply-failed | operation aborted, check logs for detail |

2020-05-25 08:46:56.802 1667 ERROR armada.cli [-] Caught unexpected exception: armada.handlers.lock.LockException: Unable to acquire lock before timeout
2020-05-25 08:46:56.802 1667 ERROR armada.cli Traceback (most recent call last):
2020-05-25 08:46:56.802 1667 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2020-05-25 08:46:56.802 1667 ERROR armada.cli self.invoke()
2020-05-25 08:46:56.802 1667 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2020-05-25 08:46:56.802 1667 ERROR armada.cli resp = self.handle(documents, tiller)
2020-05-25 08:46:56.802 1667 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 72, in func_wrapper
2020-05-25 08:46:56.802 1667 ERROR armada.cli with Lock(lock_name, bearer_token=bearer_token) as lock:
2020-05-25 08:46:56.802 1667 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 180, in __enter__
2020-05-25 08:46:56.802 1667 ERROR armada.cli self.acquire_lock()
2020-05-25 08:46:56.802 1667 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 169, in acquire_lock
2020-05-25 08:46:56.802 1667 ERROR armada.cli raise LockException("Unable to acquire lock before timeout")
2020-05-25 08:46:56.802 1667 ERROR armada.cli armada.handlers.lock.LockException: Unable to acquire lock before timeout
2020-05-25 08:46:56.802 1667 ERROR armada.cli ^[[00m

Test Activity
-------------
Sanity

[Peng Peng (ppeng)]

Log added at
https://files.starlingx.kube.cengn.ca/launchpad/1880568

[Ghada Khalil (gkhalil)]

Please monitor for a re-occurrence after the fix for https://bugs.launchpad.net/starlingx/+bug/1881353 is merged

[Ghada Khalil (gkhalil)]

I'm going to assume this is a duplicate of https://bugs.launchpad.net/starlingx/+bug/1881353
Marking as Fix Released as the fixes for the duplicate LP were merged on 2020-06-08
Please re-test