Brief Description
-----------------
apply reboot request patch on system controller of distributed cloud, patch will apply to subcould automatically. After applying is complete, there is 250.001 alarm in one DX subcloud is not cleared.
Based on Log, it looks like the runtime manifest failed to apply and the alarm was not clear.
Severity
--------
Major
Steps to Reproduce
------------------
Bring DC with 2-3 subcloud with a DX subcloud
apply a reboot reuest patch to system
After applying is completed, check alarm-list on subcloud
TC-name: DC Patching
Expected Behavior
------------------
after patching, no alarms
Actual Behavior
----------------
Reproducibility
---------------
Seen once
System Configuration
--------------------
DC system
IPv6
Subcloud5:
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+--------------------------------------------------------------------------+------------------------+----------+----------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
Brief Description
-----------------
apply reboot request patch on system controller of distributed cloud, patch will apply to subcould automatically. After applying is complete, there is 250.001 alarm in one DX subcloud is not cleared.
Based on Log, it looks like the runtime manifest failed to apply and the alarm was not clear.
Severity
--------
Major
Steps to Reproduce
------------------
Bring DC with 2-3 subcloud with a DX subcloud
apply a reboot reuest patch to system
After applying is completed, check alarm-list on subcloud
TC-name: DC Patching
Expected Behavior
------------------
after patching, no alarms
Actual Behavior
----------------
Reproducibility
---------------
Seen once
System Configuration ------- ------
-------
DC system
IPv6
Lab-name: DC Subcloud5 WCP_87-88
Branch/Pull Time/Commit ------- ------- -- 06_20-00- 00"
-------
"2019-10-
Last Pass
---------
Timestamp/Logs controller- 1 ~(keystone_admin)]$ dcmanager strategy-step list ------- -----+- ------+ ------- ------- -----+- ------- ------- ------- ------+ ------- ------- ------- ------- +------ ------- + ------- -----+- ------+ ------- ------- -----+- ------- ------- ------- ------+ ------- ------- ------- ------- +------ ------- + ------- -----+- ------+ ------- ------- -----+- ------- ------- ------- ------+ ------- ------- ------- ------- +------ ------- +
--------------
[sysadmin@
+------
| cloud | stage | state | details | started_at | finished_at |
+------
| SystemController | 1 | applying strategy | apply phase is 7% complete | 2019-10-11 13:42:12.190736 | None |
| subcloud4 | 2 | initial | | None | None |
| subcloud1 | 2 | initial | | None | None |
| subcloud5 | 2 | initial | | None | None |
+------
[sysadmin@ controller- 0 ~(keystone_admin)]$ dcmanager strategy-step list ------- -----+- ------+ ------- ---+--- ------+ ------- ------- ------- ------- +------ ------- ------- ------- -+ ------- -----+- ------+ ------- ---+--- ------+ ------- ------- ------- ------- +------ ------- ------- ------- -+ ------- -----+- ------+ ------- ---+--- ------+ ------- ------- ------- ------- +------ ------- ------- ------- -+
+------
| cloud | stage | state | details | started_at | finished_at |
+------
| SystemController | 1 | complete | | 2019-10-11 13:42:12.190736 | 2019-10-11 14:10:50.020819 |
| subcloud4 | 2 | complete | | 2019-10-11 14:11:00.026734 | 2019-10-11 14:37:04.490499 |
| subcloud1 | 2 | complete | | 2019-10-11 14:11:00.032574 | 2019-10-11 14:35:04.583524 |
| subcloud5 | 2 | complete | | 2019-10-11 14:11:00.042235 | 2019-10-11 14:38:54.640176 |
+------
Subcloud5: controller- 0 ~(keystone_admin)]$ fm alarm-list ----+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- --+---- ------- ------- ------+ ------- ---+--- ------- ------+
[sysadmin@
+------
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
| 250.001 | controller-0 Configuration is out-of-date. | host=controller-0 | major | 2019-10-11T14: |
| | | | | 26:49.619405 |
Log: conductor. manager [-] controller-1: iconfig out of date: target 6f1f9c18- 8d3d-47e2- af50-84d8a33774 00, applied 600e1267- 8fa6-4635- 8e31-b4a09aa644 07 conductor. manager [-] SYS_I Raise system config alarm: host controller-1 config applied: 600e1267- 8fa6-4635- 8e31-b4a09aa644 07 vs. target: 6f1f9c18- 8d3d-47e2- af50-84d8a33774 00. conductor. manager [-] _config_ update_ hosts config_ uuid=6f1f9c18- 8d3d-47e2- af50-84d8a33774 00 conductor. manager [-] applying runtime manifest config_ uuid=6f1f9c18- 8d3d-47e2- af50-84d8a33774 00, classes: ['openstack: :keystone: :endpoint: :runtime' , 'platform: :firewall: :runtime' , 'platform: :sysinv: :runtime' ] puppet. puppet [-] Updating hiera for host: controller-1 with config_uuid: 6f1f9c18- 8d3d-47e2- af50-84d8a33774 00 puppet. interface [-] Interface data0 has no primary address puppet. interface [-] Interface data1 has no primary address conductor. manager [-] stx-openstack app status does not warrant app re-apply apply_runtime_ manifest: fanout_cast: sending config 6f1f9c18- 8d3d-47e2- af50-84d8a33774 00 {'classes': ['openstack: :keystone: :endpoint: :runtime' , 'platform: :firewall: :runtime' , 'platform: :sysinv: :runtime' ], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'00c8462b- 9f84-4bc1- 9e49-52dac2e154 d3']} to agent agent.manager [req-a1416945- 2533-4339- be11-79682a3bec 3a admin None] config_ apply_runtime_ manifest: 6f1f9c18- 8d3d-47e2- af50-84d8a33774 00 {u'classes': [u'openstack: :keystone: :endpoint: :runtime' , u'platform: :firewall: :runtime' , u'platform: :sysinv: :runtime' ], u'force': False, u'personalities': [u'controller'], u'host_uuids': [u'00c8462b- 9f84-4bc1- 9e49-52dac2e154 d3']} controller agent.manager [req-a1416945- 2533-4339- be11-79682a3bec 3a admin None] controller-active agent.manager [req-a1416945- 2533-4339- be11-79682a3bec 3a admin None] _apply_ runtime_ manifest with hieradata_path = '/opt/platform/ puppet/ 19.10/hieradata ' api.controllers .v1.host [-] Provisioned storage node(s) [] api.controllers .v1.host [-] Provisioned storage node(s) [] api.controllers .v1.host [-] Provisioned storage node(s) [] api.controllers .v1.host [-] Provisioned storage node(s) [] puppet. common [req-a1416945- 2533-4339- be11-79682a3bec 3a admin None] Failed to execute runtime manifest for host fd01:4::4 puppet. common Traceback (most recent call last): puppet. common File "/usr/lib64/ python2. 7/site- packages/ sysinv/ puppet/ common. py", line 75, in puppet_ apply_manifest puppet. common subprocess. check_call( cmd, stdout=fnull, stderr=fnull) puppet. common File "/usr/lib64/ python2. 7/subprocess. py", line 542, in check_call puppet. common raise CalledProcessEr ror(retcode, cmd) puppet. common CalledProcessError: Command '['/usr/ local/bin/ puppet- manifest- apply.sh' , '/opt/platform/ puppet/ 19.10/hieradata ', 'fd01:4::4', 'controller', 'runtime', '/tmp/tmp1XMSjy .yaml'] ' returned non-zero exit status 1 puppet. common agent.manager [req-a1416945- 2533-4339- be11-79682a3bec 3a admin None] failed to apply runtime manifest agent.manager Traceback (most recent call last): agent.manager File "/usr/lib64/ python2. 7/site- packages/ sysinv/ agent/manager. py", line 1699, in _apply_ runtime_ manifest
2019-10-11 15:04:21.406 292409 WARNING sysinv.
2019-10-11 15:04:21.407 292409 WARNING sysinv.
2019-10-11 15:04:21.461 292409 INFO sysinv.
2019-10-11 15:04:21.461 292409 INFO sysinv.
2019-10-11 15:04:21.472 292409 INFO sysinv.
2019-10-11 15:04:22.931 292409 INFO sysinv.
2019-10-11 15:04:22.932 292409 INFO sysinv.
2019-10-11 15:04:24.633 292409 INFO sysinv.
2019-10-11 15:04:24.633 292409 INFO sysinv.agent.rpcapi [-] config_
2019-10-11 15:04:24.636 15355 INFO sysinv.
2019-10-11 15:04:24.637 15355 INFO sysinv.
2019-10-11 15:04:24.637 15355 INFO sysinv.
2019-10-11 15:04:28.167 292982 INFO sysinv.
2019-10-11 15:04:28.180 292982 INFO sysinv.
2019-10-11 15:05:34.237 292982 INFO sysinv.
2019-10-11 15:05:34.249 292982 INFO sysinv.
2019-10-11 15:05:57.081 15355 ERROR sysinv.
2019-10-11 15:05:57.081 15355 TRACE sysinv.
2019-10-11 15:05:57.081 15355 TRACE sysinv.
2019-10-11 15:05:57.081 15355 TRACE sysinv.
2019-10-11 15:05:57.081 15355 TRACE sysinv.
2019-10-11 15:05:57.081 15355 TRACE sysinv.
2019-10-11 15:05:57.081 15355 TRACE sysinv.
2019-10-11 15:05:57.081 15355 TRACE sysinv.
2019-10-11 15:05:57.084 15355 ERROR sysinv.
2019-10-11 15:05:57.084 15355 TRACE sysinv.
2019-10-11 15:05:57.084 15355 TRACE sysinv.
Test Activity
-------------
Regression Testing