| 2019-10-11 18:33:43 |
Peng Peng |
bug |
|
|
added bug |
| 2019-10-11 18:35:19 |
Peng Peng |
attachment added |
|
subcloud5-ALL_NODES_20191011.174834.tar https://bugs.launchpad.net/starlingx/+bug/1847808/+attachment/5296620/+files/subcloud5-ALL_NODES_20191011.174834.tar |
|
| 2019-10-11 18:35:46 |
Peng Peng |
attachment added |
|
system-controller-ALL_NODES_20191011.174845.tar https://bugs.launchpad.net/starlingx/+bug/1847808/+attachment/5296621/+files/system-controller-ALL_NODES_20191011.174845.tar |
|
| 2019-10-11 18:49:24 |
Peng Peng |
summary |
IPv6 Distributed Cloud: After reboot request patch applied complete, suncloud has 250.001 alarm not cleared |
IPv6 Distributed Cloud: After reboot required patch applied complete, suncloud has 250.001 alarm not cleared |
|
| 2019-10-11 18:49:51 |
Peng Peng |
description |
Brief Description
-----------------
apply reboot request patch on system controller of distributed cloud, patch will apply to subcould automatically. After applying is complete, there is 250.001 alarm in one DX subcloud is not cleared.
Based on Log, it looks like the runtime manifest failed to apply and the alarm was not clear.
Severity
--------
Major
Steps to Reproduce
------------------
Bring DC with 2-3 subcloud with a DX subcloud
apply a reboot reuest patch to system
After applying is completed, check alarm-list on subcloud
TC-name: DC Patching
Expected Behavior
------------------
after patching, no alarms
Actual Behavior
----------------
Reproducibility
---------------
Seen once
System Configuration
--------------------
DC system
IPv6
Lab-name: DC Subcloud5 WCP_87-88
Branch/Pull Time/Commit
-----------------------
"2019-10-06_20-00-00"
Last Pass
---------
Timestamp/Logs
--------------
[sysadmin@controller-1 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+-------------------+----------------------------+----------------------------+-------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+-------------------+----------------------------+----------------------------+-------------+
| SystemController | 1 | applying strategy | apply phase is 7% complete | 2019-10-11 13:42:12.190736 | None |
| subcloud4 | 2 | initial | | None | None |
| subcloud1 | 2 | initial | | None | None |
| subcloud5 | 2 | initial | | None | None |
+------------------+-------+-------------------+----------------------------+----------------------------+-------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+----------+---------+----------------------------+----------------------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+----------+---------+----------------------------+----------------------------+
| SystemController | 1 | complete | | 2019-10-11 13:42:12.190736 | 2019-10-11 14:10:50.020819 |
| subcloud4 | 2 | complete | | 2019-10-11 14:11:00.026734 | 2019-10-11 14:37:04.490499 |
| subcloud1 | 2 | complete | | 2019-10-11 14:11:00.032574 | 2019-10-11 14:35:04.583524 |
| subcloud5 | 2 | complete | | 2019-10-11 14:11:00.042235 | 2019-10-11 14:38:54.640176 |
+------------------+-------+----------+---------+----------------------------+----------------------------+
Subcloud5:
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+--------------------------------------------------------------------------+------------------------+----------+----------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
| 250.001 | controller-0 Configuration is out-of-date. | host=controller-0 | major | 2019-10-11T14: |
| | | | | 26:49.619405 |
Log:
2019-10-11 15:04:21.406 292409 WARNING sysinv.conductor.manager [-] controller-1: iconfig out of date: target 6f1f9c18-8d3d-47e2-af50-84d8a3377400, applied 600e1267-8fa6-4635-8e31-b4a09aa64407
2019-10-11 15:04:21.407 292409 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-1 config applied: 600e1267-8fa6-4635-8e31-b4a09aa64407 vs. target: 6f1f9c18-8d3d-47e2-af50-84d8a3377400.
2019-10-11 15:04:21.461 292409 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=6f1f9c18-8d3d-47e2-af50-84d8a3377400
2019-10-11 15:04:21.461 292409 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=6f1f9c18-8d3d-47e2-af50-84d8a3377400, classes: ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime', 'platform::sysinv::runtime']
2019-10-11 15:04:21.472 292409 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-1 with config_uuid: 6f1f9c18-8d3d-47e2-af50-84d8a3377400
2019-10-11 15:04:22.931 292409 INFO sysinv.puppet.interface [-] Interface data0 has no primary address
2019-10-11 15:04:22.932 292409 INFO sysinv.puppet.interface [-] Interface data1 has no primary address
2019-10-11 15:04:24.633 292409 INFO sysinv.conductor.manager [-] stx-openstack app status does not warrant app re-apply
2019-10-11 15:04:24.633 292409 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 6f1f9c18-8d3d-47e2-af50-84d8a3377400 {'classes': ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime', 'platform::sysinv::runtime'], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'00c8462b-9f84-4bc1-9e49-52dac2e154d3']} to agent
2019-10-11 15:04:24.636 15355 INFO sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] config_apply_runtime_manifest: 6f1f9c18-8d3d-47e2-af50-84d8a3377400 {u'classes': [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime', u'platform::sysinv::runtime'], u'force': False, u'personalities': [u'controller'], u'host_uuids': [u'00c8462b-9f84-4bc1-9e49-52dac2e154d3']} controller
2019-10-11 15:04:24.637 15355 INFO sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] controller-active
2019-10-11 15:04:24.637 15355 INFO sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/19.10/hieradata'
2019-10-11 15:04:28.167 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:04:28.180 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:05:34.237 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:05:34.249 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:05:57.081 15355 ERROR sysinv.puppet.common [req-a1416945-2533-4339-be11-79682a3bec3a admin None] Failed to execute runtime manifest for host fd01:4::4
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common Traceback (most recent call last):
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 75, in puppet_apply_manifest
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common subprocess.check_call(cmd, stdout=fnull, stderr=fnull)
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common raise CalledProcessError(retcode, cmd)
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/opt/platform/puppet/19.10/hieradata', 'fd01:4::4', 'controller', 'runtime', '/tmp/tmp1XMSjy.yaml']' returned non-zero exit status 1
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common
2019-10-11 15:05:57.084 15355 ERROR sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] failed to apply runtime manifest
2019-10-11 15:05:57.084 15355 TRACE sysinv.agent.manager Traceback (most recent call last):
2019-10-11 15:05:57.084 15355 TRACE sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1699, in _apply_runtime_manifest
Test Activity
-------------
Regression Testing |
Brief Description
-----------------
apply reboot required patch on system controller of distributed cloud, patch will apply to subcould automatically. After applying is complete, there is 250.001 alarm in one DX subcloud is not cleared.
Based on Log, it looks like the runtime manifest failed to apply and the alarm was not clear.
Severity
--------
Major
Steps to Reproduce
------------------
Bring DC with 2-3 subcloud with a DX subcloud
apply a reboot reuired patch to system
After applying is completed, check alarm-list on subcloud
TC-name: DC Patching
Expected Behavior
------------------
after patching, no alarms
Actual Behavior
----------------
Reproducibility
---------------
Seen once
System Configuration
--------------------
DC system
IPv6
Lab-name: DC Subcloud5 WCP_87-88
Branch/Pull Time/Commit
-----------------------
"2019-10-06_20-00-00"
Last Pass
---------
Timestamp/Logs
--------------
[sysadmin@controller-1 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+-------------------+----------------------------+----------------------------+-------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+-------------------+----------------------------+----------------------------+-------------+
| SystemController | 1 | applying strategy | apply phase is 7% complete | 2019-10-11 13:42:12.190736 | None |
| subcloud4 | 2 | initial | | None | None |
| subcloud1 | 2 | initial | | None | None |
| subcloud5 | 2 | initial | | None | None |
+------------------+-------+-------------------+----------------------------+----------------------------+-------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+----------+---------+----------------------------+----------------------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+----------+---------+----------------------------+----------------------------+
| SystemController | 1 | complete | | 2019-10-11 13:42:12.190736 | 2019-10-11 14:10:50.020819 |
| subcloud4 | 2 | complete | | 2019-10-11 14:11:00.026734 | 2019-10-11 14:37:04.490499 |
| subcloud1 | 2 | complete | | 2019-10-11 14:11:00.032574 | 2019-10-11 14:35:04.583524 |
| subcloud5 | 2 | complete | | 2019-10-11 14:11:00.042235 | 2019-10-11 14:38:54.640176 |
+------------------+-------+----------+---------+----------------------------+----------------------------+
Subcloud5:
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+--------------------------------------------------------------------------+------------------------+----------+----------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
| 250.001 | controller-0 Configuration is out-of-date. | host=controller-0 | major | 2019-10-11T14: |
| | | | | 26:49.619405 |
Log:
2019-10-11 15:04:21.406 292409 WARNING sysinv.conductor.manager [-] controller-1: iconfig out of date: target 6f1f9c18-8d3d-47e2-af50-84d8a3377400, applied 600e1267-8fa6-4635-8e31-b4a09aa64407
2019-10-11 15:04:21.407 292409 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-1 config applied: 600e1267-8fa6-4635-8e31-b4a09aa64407 vs. target: 6f1f9c18-8d3d-47e2-af50-84d8a3377400.
2019-10-11 15:04:21.461 292409 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=6f1f9c18-8d3d-47e2-af50-84d8a3377400
2019-10-11 15:04:21.461 292409 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=6f1f9c18-8d3d-47e2-af50-84d8a3377400, classes: ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime', 'platform::sysinv::runtime']
2019-10-11 15:04:21.472 292409 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-1 with config_uuid: 6f1f9c18-8d3d-47e2-af50-84d8a3377400
2019-10-11 15:04:22.931 292409 INFO sysinv.puppet.interface [-] Interface data0 has no primary address
2019-10-11 15:04:22.932 292409 INFO sysinv.puppet.interface [-] Interface data1 has no primary address
2019-10-11 15:04:24.633 292409 INFO sysinv.conductor.manager [-] stx-openstack app status does not warrant app re-apply
2019-10-11 15:04:24.633 292409 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 6f1f9c18-8d3d-47e2-af50-84d8a3377400 {'classes': ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime', 'platform::sysinv::runtime'], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'00c8462b-9f84-4bc1-9e49-52dac2e154d3']} to agent
2019-10-11 15:04:24.636 15355 INFO sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] config_apply_runtime_manifest: 6f1f9c18-8d3d-47e2-af50-84d8a3377400 {u'classes': [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime', u'platform::sysinv::runtime'], u'force': False, u'personalities': [u'controller'], u'host_uuids': [u'00c8462b-9f84-4bc1-9e49-52dac2e154d3']} controller
2019-10-11 15:04:24.637 15355 INFO sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] controller-active
2019-10-11 15:04:24.637 15355 INFO sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/19.10/hieradata'
2019-10-11 15:04:28.167 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:04:28.180 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:05:34.237 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:05:34.249 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:05:57.081 15355 ERROR sysinv.puppet.common [req-a1416945-2533-4339-be11-79682a3bec3a admin None] Failed to execute runtime manifest for host fd01:4::4
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common Traceback (most recent call last):
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 75, in puppet_apply_manifest
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common subprocess.check_call(cmd, stdout=fnull, stderr=fnull)
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common raise CalledProcessError(retcode, cmd)
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/opt/platform/puppet/19.10/hieradata', 'fd01:4::4', 'controller', 'runtime', '/tmp/tmp1XMSjy.yaml']' returned non-zero exit status 1
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common
2019-10-11 15:05:57.084 15355 ERROR sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] failed to apply runtime manifest
2019-10-11 15:05:57.084 15355 TRACE sysinv.agent.manager Traceback (most recent call last):
2019-10-11 15:05:57.084 15355 TRACE sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1699, in _apply_runtime_manifest
Test Activity
-------------
Regression Testing |
|
| 2019-10-11 19:09:32 |
Ghada Khalil |
summary |
IPv6 Distributed Cloud: After reboot required patch applied complete, suncloud has 250.001 alarm not cleared |
IPv6 Distributed Cloud: After reboot required patch applied complete, subcloud has 250.001 alarm not cleared |
|
| 2019-10-11 19:10:04 |
Ghada Khalil |
tags |
|
stx.distcloud |
|
| 2019-10-11 19:11:37 |
Ghada Khalil |
description |
Brief Description
-----------------
apply reboot required patch on system controller of distributed cloud, patch will apply to subcould automatically. After applying is complete, there is 250.001 alarm in one DX subcloud is not cleared.
Based on Log, it looks like the runtime manifest failed to apply and the alarm was not clear.
Severity
--------
Major
Steps to Reproduce
------------------
Bring DC with 2-3 subcloud with a DX subcloud
apply a reboot reuired patch to system
After applying is completed, check alarm-list on subcloud
TC-name: DC Patching
Expected Behavior
------------------
after patching, no alarms
Actual Behavior
----------------
Reproducibility
---------------
Seen once
System Configuration
--------------------
DC system
IPv6
Lab-name: DC Subcloud5 WCP_87-88
Branch/Pull Time/Commit
-----------------------
"2019-10-06_20-00-00"
Last Pass
---------
Timestamp/Logs
--------------
[sysadmin@controller-1 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+-------------------+----------------------------+----------------------------+-------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+-------------------+----------------------------+----------------------------+-------------+
| SystemController | 1 | applying strategy | apply phase is 7% complete | 2019-10-11 13:42:12.190736 | None |
| subcloud4 | 2 | initial | | None | None |
| subcloud1 | 2 | initial | | None | None |
| subcloud5 | 2 | initial | | None | None |
+------------------+-------+-------------------+----------------------------+----------------------------+-------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+----------+---------+----------------------------+----------------------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+----------+---------+----------------------------+----------------------------+
| SystemController | 1 | complete | | 2019-10-11 13:42:12.190736 | 2019-10-11 14:10:50.020819 |
| subcloud4 | 2 | complete | | 2019-10-11 14:11:00.026734 | 2019-10-11 14:37:04.490499 |
| subcloud1 | 2 | complete | | 2019-10-11 14:11:00.032574 | 2019-10-11 14:35:04.583524 |
| subcloud5 | 2 | complete | | 2019-10-11 14:11:00.042235 | 2019-10-11 14:38:54.640176 |
+------------------+-------+----------+---------+----------------------------+----------------------------+
Subcloud5:
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+--------------------------------------------------------------------------+------------------------+----------+----------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
| 250.001 | controller-0 Configuration is out-of-date. | host=controller-0 | major | 2019-10-11T14: |
| | | | | 26:49.619405 |
Log:
2019-10-11 15:04:21.406 292409 WARNING sysinv.conductor.manager [-] controller-1: iconfig out of date: target 6f1f9c18-8d3d-47e2-af50-84d8a3377400, applied 600e1267-8fa6-4635-8e31-b4a09aa64407
2019-10-11 15:04:21.407 292409 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-1 config applied: 600e1267-8fa6-4635-8e31-b4a09aa64407 vs. target: 6f1f9c18-8d3d-47e2-af50-84d8a3377400.
2019-10-11 15:04:21.461 292409 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=6f1f9c18-8d3d-47e2-af50-84d8a3377400
2019-10-11 15:04:21.461 292409 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=6f1f9c18-8d3d-47e2-af50-84d8a3377400, classes: ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime', 'platform::sysinv::runtime']
2019-10-11 15:04:21.472 292409 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-1 with config_uuid: 6f1f9c18-8d3d-47e2-af50-84d8a3377400
2019-10-11 15:04:22.931 292409 INFO sysinv.puppet.interface [-] Interface data0 has no primary address
2019-10-11 15:04:22.932 292409 INFO sysinv.puppet.interface [-] Interface data1 has no primary address
2019-10-11 15:04:24.633 292409 INFO sysinv.conductor.manager [-] stx-openstack app status does not warrant app re-apply
2019-10-11 15:04:24.633 292409 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 6f1f9c18-8d3d-47e2-af50-84d8a3377400 {'classes': ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime', 'platform::sysinv::runtime'], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'00c8462b-9f84-4bc1-9e49-52dac2e154d3']} to agent
2019-10-11 15:04:24.636 15355 INFO sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] config_apply_runtime_manifest: 6f1f9c18-8d3d-47e2-af50-84d8a3377400 {u'classes': [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime', u'platform::sysinv::runtime'], u'force': False, u'personalities': [u'controller'], u'host_uuids': [u'00c8462b-9f84-4bc1-9e49-52dac2e154d3']} controller
2019-10-11 15:04:24.637 15355 INFO sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] controller-active
2019-10-11 15:04:24.637 15355 INFO sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/19.10/hieradata'
2019-10-11 15:04:28.167 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:04:28.180 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:05:34.237 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:05:34.249 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:05:57.081 15355 ERROR sysinv.puppet.common [req-a1416945-2533-4339-be11-79682a3bec3a admin None] Failed to execute runtime manifest for host fd01:4::4
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common Traceback (most recent call last):
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 75, in puppet_apply_manifest
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common subprocess.check_call(cmd, stdout=fnull, stderr=fnull)
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common raise CalledProcessError(retcode, cmd)
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/opt/platform/puppet/19.10/hieradata', 'fd01:4::4', 'controller', 'runtime', '/tmp/tmp1XMSjy.yaml']' returned non-zero exit status 1
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common
2019-10-11 15:05:57.084 15355 ERROR sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] failed to apply runtime manifest
2019-10-11 15:05:57.084 15355 TRACE sysinv.agent.manager Traceback (most recent call last):
2019-10-11 15:05:57.084 15355 TRACE sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1699, in _apply_runtime_manifest
Test Activity
-------------
Regression Testing |
Brief Description
-----------------
apply reboot required patch on system controller of distributed cloud, patch will apply to subcould automatically. After applying is complete, there is 250.001 alarm in one DX subcloud is not cleared.
Based on Log, it looks like the runtime manifest failed to apply and the alarm was not clear.
Severity
--------
Major
Steps to Reproduce
------------------
Bring DC with 2-3 subcloud with a DX subcloud
apply a reboot reuired patch to system
After applying is completed, check alarm-list on subcloud
TC-name: DC Patching
Expected Behavior
------------------
after patching, no alarms
Actual Behavior
----------------
Reproducibility
---------------
Tried once; not sure
System Configuration
--------------------
DC system
IPv6
Lab-name: DC Subcloud5 WCP_87-88
Branch/Pull Time/Commit
-----------------------
"2019-10-06_20-00-00"
Last Pass
---------
Timestamp/Logs
--------------
[sysadmin@controller-1 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+-------------------+----------------------------+----------------------------+-------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+-------------------+----------------------------+----------------------------+-------------+
| SystemController | 1 | applying strategy | apply phase is 7% complete | 2019-10-11 13:42:12.190736 | None |
| subcloud4 | 2 | initial | | None | None |
| subcloud1 | 2 | initial | | None | None |
| subcloud5 | 2 | initial | | None | None |
+------------------+-------+-------------------+----------------------------+----------------------------+-------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+----------+---------+----------------------------+----------------------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+----------+---------+----------------------------+----------------------------+
| SystemController | 1 | complete | | 2019-10-11 13:42:12.190736 | 2019-10-11 14:10:50.020819 |
| subcloud4 | 2 | complete | | 2019-10-11 14:11:00.026734 | 2019-10-11 14:37:04.490499 |
| subcloud1 | 2 | complete | | 2019-10-11 14:11:00.032574 | 2019-10-11 14:35:04.583524 |
| subcloud5 | 2 | complete | | 2019-10-11 14:11:00.042235 | 2019-10-11 14:38:54.640176 |
+------------------+-------+----------+---------+----------------------------+----------------------------+
Subcloud5:
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+--------------------------------------------------------------------------+------------------------+----------+----------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
| 250.001 | controller-0 Configuration is out-of-date. | host=controller-0 | major | 2019-10-11T14: |
| | | | | 26:49.619405 |
Log:
2019-10-11 15:04:21.406 292409 WARNING sysinv.conductor.manager [-] controller-1: iconfig out of date: target 6f1f9c18-8d3d-47e2-af50-84d8a3377400, applied 600e1267-8fa6-4635-8e31-b4a09aa64407
2019-10-11 15:04:21.407 292409 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-1 config applied: 600e1267-8fa6-4635-8e31-b4a09aa64407 vs. target: 6f1f9c18-8d3d-47e2-af50-84d8a3377400.
2019-10-11 15:04:21.461 292409 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=6f1f9c18-8d3d-47e2-af50-84d8a3377400
2019-10-11 15:04:21.461 292409 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=6f1f9c18-8d3d-47e2-af50-84d8a3377400, classes: ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime', 'platform::sysinv::runtime']
2019-10-11 15:04:21.472 292409 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-1 with config_uuid: 6f1f9c18-8d3d-47e2-af50-84d8a3377400
2019-10-11 15:04:22.931 292409 INFO sysinv.puppet.interface [-] Interface data0 has no primary address
2019-10-11 15:04:22.932 292409 INFO sysinv.puppet.interface [-] Interface data1 has no primary address
2019-10-11 15:04:24.633 292409 INFO sysinv.conductor.manager [-] stx-openstack app status does not warrant app re-apply
2019-10-11 15:04:24.633 292409 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 6f1f9c18-8d3d-47e2-af50-84d8a3377400 {'classes': ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime', 'platform::sysinv::runtime'], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'00c8462b-9f84-4bc1-9e49-52dac2e154d3']} to agent
2019-10-11 15:04:24.636 15355 INFO sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] config_apply_runtime_manifest: 6f1f9c18-8d3d-47e2-af50-84d8a3377400 {u'classes': [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime', u'platform::sysinv::runtime'], u'force': False, u'personalities': [u'controller'], u'host_uuids': [u'00c8462b-9f84-4bc1-9e49-52dac2e154d3']} controller
2019-10-11 15:04:24.637 15355 INFO sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] controller-active
2019-10-11 15:04:24.637 15355 INFO sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/19.10/hieradata'
2019-10-11 15:04:28.167 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:04:28.180 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:05:34.237 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:05:34.249 292982 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-11 15:05:57.081 15355 ERROR sysinv.puppet.common [req-a1416945-2533-4339-be11-79682a3bec3a admin None] Failed to execute runtime manifest for host fd01:4::4
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common Traceback (most recent call last):
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common File "/usr/lib64/python2.7/site-packages/sysinv/puppet/common.py", line 75, in puppet_apply_manifest
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common subprocess.check_call(cmd, stdout=fnull, stderr=fnull)
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common raise CalledProcessError(retcode, cmd)
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh', '/opt/platform/puppet/19.10/hieradata', 'fd01:4::4', 'controller', 'runtime', '/tmp/tmp1XMSjy.yaml']' returned non-zero exit status 1
2019-10-11 15:05:57.081 15355 TRACE sysinv.puppet.common
2019-10-11 15:05:57.084 15355 ERROR sysinv.agent.manager [req-a1416945-2533-4339-be11-79682a3bec3a admin None] failed to apply runtime manifest
2019-10-11 15:05:57.084 15355 TRACE sysinv.agent.manager Traceback (most recent call last):
2019-10-11 15:05:57.084 15355 TRACE sysinv.agent.manager File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1699, in _apply_runtime_manifest
Test Activity
-------------
Regression Testing |
|
| 2019-10-11 19:11:53 |
Ghada Khalil |
tags |
stx.distcloud |
stx.distcloud stx.update |
|
| 2019-10-11 19:12:31 |
Ghada Khalil |
starlingx: assignee |
|
Don Penney (dpenney) |
|
| 2019-10-11 19:35:23 |
Ghada Khalil |
starlingx: assignee |
Don Penney (dpenney) |
Andy (andy.wrs) |
|
| 2019-10-11 21:25:04 |
Ghada Khalil |
tags |
stx.distcloud stx.update |
stx.distcloud |
|
| 2019-10-15 14:38:33 |
Ghada Khalil |
bug |
|
|
added subscriber Bill Zvonar |
| 2019-10-16 17:19:45 |
Yang Liu |
tags |
stx.distcloud |
stx.distcloud stx.retestneeded |
|
| 2019-10-16 17:50:49 |
Ghada Khalil |
tags |
stx.distcloud stx.retestneeded |
stx.3.0 stx.distcloud stx.retestneeded |
|
| 2019-10-16 17:50:55 |
Ghada Khalil |
starlingx: status |
New |
Triaged |
|
| 2019-10-16 17:50:59 |
Ghada Khalil |
starlingx: importance |
Undecided |
Medium |
|
| 2019-10-17 14:46:14 |
Ghada Khalil |
starlingx: status |
Triaged |
In Progress |
|
| 2019-10-21 12:32:45 |
OpenStack Infra |
starlingx: status |
In Progress |
Fix Released |
|
| 2019-11-06 14:46:47 |
Peng Peng |
tags |
stx.3.0 stx.distcloud stx.retestneeded |
stx.3.0 stx.distcloud |
|
