Re-apply of application failed if the previous Armada apply failed with an exception
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Medium
|
Angie Wang |
Bug Description
Brief Description
-----------------
If the apply/remove of an application is failed due to an exception of the Armada operation or an abnormal exit of the Armada operation, the Armada lock cannot be released which causes the subsequent re-apply of the application to fail as it cannot acquire the lock.
Severity
--------
Major
Steps to Reproduce
------------------
The exception of armada operation may be caused by the connection issue(ie. lose connection to the k8s cluster/connection aborted by the k8s cluster)
To simulate,
1. Applying an application
2. Terminate the armada pod during armada apply
Expected Behavior
------------------
The armada lock is released and subsequent re-apply works.
Actual Behavior
----------------
The armada lock is not released and subsequent re-apply fails.
System Configuration
-------
Any types of system
Last Pass
---------
N/A
Timestamp/Logs
--------------
The first apply of platform-
sysinv 2020-07-29 15:33:28.025 101299 ERROR sysinv.
sysinv 2020-07-29 15:33:28.027 101299 INFO sysinv.
The re-try of platform-
sysinv 2020-07-29 15:33:28.028 101299 INFO sysinv.
2020-07-29 15:33:59.566 37 WARNING armada.
2020-07-29 15:34:59.796 37 ERROR armada.cli [-] Caught unexpected exception: armada.
2020-07-29 15:34:59.796 37 ERROR armada.cli Traceback (most recent call last):
2020-07-29 15:34:59.796 37 ERROR armada.cli File "/usr/local/
2020-07-29 15:34:59.796 37 ERROR armada.cli self.invoke()
2020-07-29 15:34:59.796 37 ERROR armada.cli File "/usr/local/
2020-07-29 15:34:59.796 37 ERROR armada.cli resp = self.handle(
2020-07-29 15:34:59.796 37 ERROR armada.cli File "/usr/local/
2020-07-29 15:34:59.796 37 ERROR armada.cli with Lock(lock_name, bearer_
2020-07-29 15:34:59.796 37 ERROR armada.cli File "/usr/local/
2020-07-29 15:34:59.796 37 ERROR armada.cli self.acquire_lock()
2020-07-29 15:34:59.796 37 ERROR armada.cli File "/usr/local/
2020-07-29 15:34:59.796 37 ERROR armada.cli raise LockException(
2020-07-29 15:34:59.796 37 ERROR armada.cli armada.
2020-07-29 15:34:59.796 37 ERROR armada.cli ^[[00m
Test Activity
-------------
Developer Testing
Workaround
----------
kubectl delete locks.armada.
Changed in starlingx: | |
assignee: | nobody → Angie Wang (angiewang) |
summary: |
- Re-apply of application failed if the previous apply failed with an - exception + Re-apply of application failed if the previous Armada apply failed with + an exception |
Changed in starlingx: | |
importance: | Undecided → Medium |
status: | New → Triaged |
tags: | added: stx.5.0 stx.config stx.containers |
Changed in starlingx: | |
status: | Triaged → In Progress |
Fix up for review: https:/ /review. opendev. org/c/starlingx /config/ +/770482