swact active controller rejected due to config-out-of-date alarm

[Yvonne Ding]

Brief Description
-----------------
Swact active controller is failed after cold migration. The config of standby controller is not applied.

Severity
--------
Major

Steps to Reproduce
------------------
1. Create a flavor with 1 vcpu
2. Add following extra specs: {'hw:cpu_policy': 'shared'}
3. Create a volume from ubuntu_14 image
4. Boot a ubuntu_14 VM with above flavor from volume
5. Live migrate ubuntu_14 VM
6. Ping vm from NatBox after live migration
7. Cold migrate vm and check vm is moved to different host
8. Ping vm from NatBox after cold migration
9. Swact active controller

TC-name:
test_migrate_vms.py::test_migrate_vm_various_guest

Expected Behavior
-----------------
Swact active controller succeed

Actual Behavior
----------------
Swact active controller failed

Reproducibility
---------------
reproducible

System Configuration
--------------------
Regular standard 2+2

Lab-name:
ip_1_4

Branch/Pull Time/Commit
-----------------------
BUILD_ID="20210226T024233Z"

Timestamp/Logs
--------------
[2021-03-07 07:02:46,315] 1651 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4
[2021-03-07 07:02:46,315] 479 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2021-03-07 07:02:46,315] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-swact controller-0'
[2021-03-07 07:02:47,684] 436 DEBUG MainThread ssh.expect :: Output:
controller-1 target Config 84ef0ae4-fc54-415d-9476-49c74b1f0135 not yet applied. Apply target Config via Lock/Unlock prior to Swact

logs of .tar and TIS_AUTOMATION.log as below,
https://files.starlingx.kube.cengn.ca/launchpad/
Note: above url is currently not accessible

Test Activity
-------------
STX Nova regression

[Yang Liu (yliu12)]

The swact rejection is expected from software perspective when standby controller is config out-of-date.

Please look at automation logs to find out when the config out-of-date alarm appeared and stuck. The test case that caused this stuck alarm will need to be investigated instead.

[Yvonne Ding (yding)]

Please note above .tar and .log will be moved to cengn log portal when it is back to up. Currently cengn log portal is unavailable.

[Yvonne Ding (yding)]

The standby controller config out-of-date alarm appeared at gathering system health info before test session begins. The alarm is not caused by testcase. --> "20210306T023358Z" load.

[Ghada Khalil (gkhalil)]

This is an issue where a config-out-of-date alarm was present on the system, resulting on the swact getting rejected. There has been a fix for the config-of-date alarm condition in a later load.
https://bugs.launchpad.net/starlingx/+bug/1915343
Fixed by https://opendev.org/starlingx/config/commit/df6a3386544a2c1defcf4a9cc35f3eda0daa1e4f on 2021-03-24

[Ghada Khalil (gkhalil)]

Marking as a duplicate of https://bugs.launchpad.net/starlingx/+bug/1915343