Configuration is out-of-date alarm was not cleared after SX host lock/unlock
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Medium
|
Stefan Dinescu |
Bug Description
Brief Description
-----------------
On SX system, after host lock/unlock, 250.001 | controller-0 Configuration is out-of-date alarm raised and not cleared in 10 mins.
The alarm was cleared after another host lock/unlock.
Severity
--------
Major
Steps to Reproduce
------------------
host lock/unlock
check alarm-list
TC-name: test_lock_
Expected Behavior
------------------
no 250.001 alarm
Actual Behavior
----------------
250.001 not cleared
Reproducibility
---------------
Seen once
System Configuration
-------
One node system
Lab-name: SM-3
Branch/Pull Time/Commit
-------
stx master as of 2019-08-26_20-59-00
Last Pass
---------
2019-08-24_20-59-00
Timestamp/Logs
--------------
[2019-08-27 08:04:52,217] 301 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-08-27 08:04:54,343] 423 DEBUG MainThread ssh.expect :: Output:
+------
| UUID | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+------
| ea1dbc10-
+------
[2019-08-27 08:04:59,419] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-08-27 08:05:51,293] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-08-27 08:27:36,096] 301 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-08-27 08:27:38,150] 423 DEBUG MainThread ssh.expect :: Output:
+------
| UUID | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+------
| 06925597-
[2019-08-27 08:37:30,775] 301 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-08-27 08:37:32,723] 423 DEBUG MainThread ssh.expect :: Output:
+------
| UUID | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+------
| 06925597-
[2019-08-27 08:37:55,560] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-08-27 08:38:45,705] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-08-27 08:53:55,523] 301 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-08-27 08:53:57,643] 423 DEBUG MainThread ssh.expect :: Output:
+------
| UUID | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+------
| fbeb4356-
| e68e85c9-
| 03ab993f-
| 98cc5502-
+------
controller-0:~$
Test Activity
-------------
Sanity
tags: | added: stx.retestneeded |
Changed in starlingx: | |
assignee: | nobody → Stefan Dinescu (stefandinescu) |
importance: | Undecided → Medium |
tags: | added: stx.3.0 stx.config |
Changed in starlingx: | |
status: | New → Triaged |
The stuck alarm occurs near this log
2019-08-27 08:15:31.123 95305 INFO sysinv.agent.rpcapi [-] config_ apply_runtime_ manifest: fanout_cast: sending config 3ac0c536- 57ad-4e49- ba37-6877ef0306 22 {'classes': ['openstack: :keystone: :endpoint: :runtime' , 'platform: :firewall: :runtime' , 'platform: :sysinv: :runtime' ], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'94146b1f- 8449-4cd8- 888b-0cffa07293 f8']} to agent openstack. common. rpc.common [-] Connected to AMQP server on 192.168.204.2:5672 conductor. manager [-] drbd-overview: pgsql-40.0, cgcs-20.0, extension-0.96875, patch-vault-0, etcd-4.8, dockerdistribut ion-16. 0 conductor. manager [-] lvdisplay: pgsql-40.0, cgcs-20.0, extension-1.0, patch-vault-0, etcd-5.0, dockerdistribut ion-16. 0 conductor. manager [-] SYS_I Clear system config alarm: controller-0 target config 328f0cd7- a654-4419- 8320-18f619dc2e a4 agent.manager [-] get_ihost_by_macs rpc Timeout. openstack. common. rpc.common [-] Connected to AMQP server on 192.168.204.2:5672 openstack. common. loopingcall [-] task run outlasted interval by 1.728777 sec
2019-08-27 08:15:31.132 95305 INFO sysinv.
2019-08-27 08:15:31.223 95305 INFO sysinv.
2019-08-27 08:15:31.223 95305 INFO sysinv.
2019-08-27 08:15:31.235 95305 INFO sysinv.
2019-08-27 08:16:03.390 14322 INFO sysinv.
2019-08-27 08:16:03.399 14322 INFO sysinv.
2019-08-27 08:16:03.417 14322 WARNING sysinv.
We can determine what area of the code is being run, but not why there is no puppet log for that time period.