Alarm "Configuration is out-of-date" is not cleared after service http/https port change

[ayyappa]

Brief Description
-----------------
Alarm "Configuration is out-of-date" is not cleared after modify,apply service http to a new port

Severity
--------
Minor

Steps to Reproduce
------------------
1)Modify the service http to new port 8887 with the followig command
system service-parameter-modify http config http_port=8887

2)Apply the new config with the following command
system service-parameter-apply http

[sysadmin@controller-0 ~(keystone_admin)]$ system service-parameter-list --service http
+--------------------------------------+---------+---------+--------+-------+-------------+----------+
| uuid | service | section | name | value | personality | resource |
+--------------------------------------+---------+---------+--------+-------+-------------+----------+
| ca737769-472e-4de7-bb87-a65ac9d86b48 | http | config | http_p | 8887 | None | None |
| | | | ort | | | |
| | | | | | | |
| 601b6a05-52e5-4290-b21d-5dcddef1e61d | http | config | https_ | 8443 | None | None |
| | | | port | | | |
| | | | | | | |

3)verify the alarms

[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+--------------------------------------------+----------------+----------+-------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+--------------------------------------------+----------------+----------+-------------+
| 250.001 | controller-0 Configuration is out-of-date. | host= | major | 2020-01-10T |
| | | controller-0 | | 18:24:05. |
| | | | | 489544 |
| | | | | |

Expected Behavior
------------------
The alarm "Configuration is out-of-date" should be cleared after config applied on all the hosts

Actual Behavior
----------------
The alarm "Configuration is out-of-date" is not cleared and stays forever

Reproducibility
---------------
100%

System Configuration
--------------------
tested on all the following systems
one node system,http,ipv4
standard system,http,ipv4
one node system,http,ipv6

Branch/Pull Time/Commit
-----------------------
2020-01-09

Last Pass
---------
may,2019

Timestamp/Logs
--------------
2020-01-10T18:24:05.489544

Test Activity
-------------
Feature Automation

Workaround
----------
Delete the alarm manually by "fm alarm-delete <uuid>"

[Ghada Khalil (gkhalil)]

stx.4.0 / medium priority - stale alarm / workaround is to clear it manually

[Chris Winnicki (chriswinnicki)]

This issue has been seen again during automated Networking Regression in the following load / Wind River lab:

Lab: WCP_71_75
Load: 2020-04-03_00-10-00
Lab Config: 2+3

Failure signature is:
====================== Test Step 3: Modify the service to new port number
[2020-04-04 14:55:32,069] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[face::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne service-parameter-modify http config http_port=8887'

[2020-04-04 14:55:35,137] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[face::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne service-parameter-apply http'

The following alarms persist after config change:

[2020-04-04 15:00:52,714] 436 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+----------+--------------------------------------------+-------------------+----------+----------------------------+
| UUID | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+--------------------------------------+----------+--------------------------------------------+-------------------+----------+----------------------------+
| 4498022c-3b1a-4712-8b98-402fbf05f020 | 250.001 | controller-1 Configuration is out-of-date. | host=controller-1 | major | 2020-04-04T14:55:36.704585 |
| eca11d0e-d967-46e1-b21d-eb6788e1fc3e | 250.001 | controller-0 Configuration is out-of-date. | host=controller-0 | major | 2020-04-04T14:55:36.367628 |
+--------------------------------------+----------+--------------------------------------------+-------------------+----------+----------------------------+

* New logs attached

[Ghada Khalil (gkhalil)]

This looks like a duplicate of https://bugs.launchpad.net/starlingx/+bug/1873600
Assigning to Andy to confirm

[Andy (andy.wrs)]

Yes it is a duplicate of https://bugs.launchpad.net/starlingx/+bug/1873600 and should be fixed by the same fix.

[Ghada Khalil (gkhalil)]

Marking as a duplicate and setting the state to Fix Released

Fixed by: https://review.opendev.org/#/c/730138/
Merged on 2020-05-26

[Difu Hu (difuhu)]

This issue is not seen with latest fix.

[Chris Winnicki (chriswinnicki)]

Reopening the defect
"Configuration is out-of-date" alarms still present after modifying
http_port from 8080 to 8887

Build: 2020-05-29_20-00-00

Alarms:
+--------------------------------------+----------+--------------------------------------------+-------------------+----------+----------------------------+
| UUID | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+--------------------------------------+----------+--------------------------------------------+-------------------+----------+----------------------------+
| 15c81d49-22f3-4ba3-af46-4b248ba415f4 | 250.001 | controller-1 Configuration is out-of-date. | host=controller-1 | major | 2020-05-30T17:26:29.087980 |
| c6500edb-c215-4433-9cfd-5af48d3265ae | 250.001 | controller-0 Configuration is out-of-date. | host=controller-0 | major | 2020-05-30T17:26:28.844582 |
+--------------------------------------+----------+--------------------------------------------+-------------------+----------+----------------------------+

Fresh set of logs is attached

[Difu Hu (difuhu)]

Sorry, previous close action is a mistake.
I intended to close another LP.

[Ghada Khalil (gkhalil)]

@Chris Winnicki, the duplicate LP was re-opened. The follow-up fix will be tracked there.

[Ghada Khalil (gkhalil)]

Additional fix merged for the duplicate LP
https://review.opendev.org/733164
Merged on 2020-06-09

[Yang Liu (yliu12)]

Test passed on:
Lab: WCP_71_75
Load: 2020-06-12_20-00-00