StarlingX

controller-0 experienced a configuration failure after active controller reboot

Bug #1891658 reported by Peng Peng on 2020-08-14

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	Invalid	Medium	Paul-Ionut Vaduva

Bug Description

Brief Description
-----------------
In SX system, after controller force reboot, the status of controller became "avaialble" in a very short period, then stuck at "degraded". And 200.011 alarm " controller-0 experienced a configuration failure" raised.

Severity
--------
Major

Steps to Reproduce
------------------
reboot -f active cpntroller

TC-name: mtc/test_ungraceful_reboot.py::test_force_reboot_host[controller]

Expected Behavior
------------------
controller status become avaiable after reboot

Actual Behavior
----------------
controller status stuck at "degraded" after reboot

Reproducibility
---------------
Unknown - first time this is seen in sanity

System Configuration
--------------------
One node system

Lab-name: WCP_112

Branch/Pull Time/Commit
-----------------------
2020-08-13_20-00-00

Last Pass
---------
2020-08-11_20-00-00

Timestamp/Logs
--------------
[2020-08-14 04:10:07,782] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-show controller-0'
[2020-08-14 04:10:09,370] 436 DEBUG MainThread ssh.expect :: Output:
+-----------------------+----------------------------------------------------------------------+
| Property | Value |
+-----------------------+----------------------------------------------------------------------+
| action | none |
| administrative | unlocked |
| availability | available |

[2020-08-14 04:16:41,485] 314 DEBUG MainThread ssh.send :: Send 'sudo reboot -f'

[2020-08-14 04:38:42,016] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne show'
[2020-08-14 04:38:44,912] 436 DEBUG MainThread ssh.expect :: Output:
[Errno 111] Connection refused

[2020-08-14 04:39:09,370] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-show controller-0'
[2020-08-14 04:39:10,896] 436 DEBUG MainThread ssh.expect :: Output:
+-----------------------+----------------------------------------------------------------------+
| Property | Value |
+-----------------------+----------------------------------------------------------------------+
| action | none |
| administrative | unlocked |
| availability | available |

[2020-08-14 04:39:22,645] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-show controller-0'
[2020-08-14 04:39:24,156] 436 DEBUG MainThread ssh.expect :: Output:
+-----------------------+----------------------------------------------------------------------+
| Property | Value |
+-----------------------+----------------------------------------------------------------------+
| action | none |
| administrative | unlocked |
| availability | degraded |

[2020-08-14 05:50:37,247] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-show controller-0'
[2020-08-14 05:50:38,869] 436 DEBUG MainThread ssh.expect :: Output:
+-----------------------+----------------------------------------------------------------------+
| Property | Value |
+-----------------------+----------------------------------------------------------------------+
| action | none |
| administrative | unlocked |
| availability | degraded |

2020-08-14 05:56:50,214] 314 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne alarm-list --nowrap --uuid'
[2020-08-14 05:56:51,834] 436 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+----------+----------------------------------------------------+-------------------+----------+----------------------------+
| UUID | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+--------------------------------------+----------+----------------------------------------------------+-------------------+----------+----------------------------+
| bbdf4e6d-2329-44f6-8a1c-72abf1208cca | 200.011 | controller-0 experienced a configuration failure. | host=controller-0 | critical | 2020-08-14T04:39:22.763330 |
+--------------------------------------+----------+----------------------------------------------------+-------------------+----------+----------------------------+

Test Activity
-------------
Sanity

Tags:

Revision history for this message

Peng Peng (ppeng) wrote on 2020-08-14:

collect log added at:
https://files.starlingx.kube.cengn.ca/launchpad/1891658

tags:

added: stx.retestneeded

Revision history for this message

Difu Hu (difuhu) wrote on 2020-08-28:

Similar issue occurred on AIO-DX wp_13_14 build 2020-08-27_20-00-00
After initial installation controller-1 was in "degraded" status, and 200.011 alarm " controller-0 experienced a configuration failure" raised.
log: https://files.starlingx.kube.cengn.ca/launchpad/1891658/ wp_13_14_ALL_NODES_20200828.133937.tar

+--------------------------------------+----------+----------------------------------------------------+-------------------+----------+----------------------------+
| UUID | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+--------------------------------------+----------+----------------------------------------------------+-------------------+----------+----------------------------+
| deba472a-6437-480c-86a3-0846993e675f | 200.011 | controller-1 experienced a configuration failure. | host=controller-1 | critical | 2020-08-28T03:44:03.261544 |
+--------------------------------------+----------+----------------------------------------------------+-------------------+----------+----------------------------+

Ghada Khalil (gkhalil) on 2020-09-09

tags:

added: stx.config

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2020-10-19:

This likely a failure in puppet/sysinv. Since the initial occurrences in August, it's not clear if this was seen again.

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2020-10-19:

Marking as stx.5.0 / medium priority - logs should be reviewed to determine what failed in the two cases above

tags:	added: stx.5.0
Changed in starlingx:
importance:	Undecided → Medium
status:	New → Triaged
assignee:	nobody → Paul-Ionut Vaduva (pvaduva)

Revision history for this message

Paul-Ionut Vaduva (pvaduva) wrote on 2020-12-08:

Download full text (31.4 KiB)

Boot
2020-08-14T03:01:25.255 localhost kernel: notice [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1127.el7.4.tis.x86_64 root=UUID=170af5d4-98ab-4909-b0da-dd83f3da53dc ro module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto console=ttyS0,115200n8 iommu=pt usbcore.autosuspend=-1 hugepagesz=2M hugepages=0 default_hugepagesz=2M rcu_nocbs=2-43 kthread_cpus=0,1 irqaffinity=0,1 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1 intel_iommu=on biosdevname=0 kvm-intel.eptad=0 user_namespace.enable=1

Boot 
2020-08-14T03:01:25.255 localhost kernel: notice [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1127.el7.4.tis.x86_64 root=UUID=170af5d4-98ab-4909-b0da-dd83f3da53dc ro module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto console=ttyS0,115200n8 iommu=pt usbcore.autosuspend=-1 hugepagesz=2M hugepages=0 default_hugepagesz=2M rcu_nocbs=2-43 kthread_cpus=0,1 irqaffinity=0,1 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1 intel_iommu=on biosdevname=0 kvm-intel.eptad=0 user_namespace.enable=1

Runtime manifests
2020-08-14 03:11:16.861 82634 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=674e7e14-de1c-426b-82dc-43ba718d555b
2020-08-14 03:11:16.882 82634 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 674e7e14-de1c-426b-82dc-43ba718d555b {'force': False, 'personalities': ['worker', 'storage']} to agent
2020-08-14 03:11:16.893 82634 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=70559c72-155b-47f0-8217-83aa7e8d5bd1, classes: ['openstack::horizon::runtime']
2020-08-14 03:11:16.912 82634 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 70559c72-155b-47f0-8217-83aa7e8d5bd1 {'classes': ['openstack::horizon::runtime'], 'force': False, 'personalities': ['controller']} to agent
2020-08-14 03:11:29.353 82634 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=28b15c54-9267-4edd-8c54-e3cd212e8dea, classes: ['platform::compute::grub::runtime', 'platform::compute::config::runtime']
2020-08-14 03:11:29.372 82634 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 28b15c54-9267-4edd-8c54-e3cd212e8dea {'classes': ['platform::compute::grub::runtime', 'platform::compute::config::runtime'], 'force': False, 'personalities': ['controller', 'worker'], 'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} to agent
2020-08-14 03:11:33.847 82634 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=1621013d-1648-49fa-9293-1d2561a8d419, classes: ['openstack::keystone::endpoint::runtime', 'openstack::barbican::runtime']
2020-08-14 03:11:36.530 82634 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 1621013d-1648-49fa-9293-1d2561a8d419 {'classes': ['openstack::keystone::endpoint::runtime', 'openstack::barbican::runtime'], 'force': True, 'personalities': ['controller'], 'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} to agent
2020-08-14 03:11:36.535 75323 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 1621013d-1648-49fa-9293-1d2561a8d419 {u'classes': [u'openstack::keystone::endpoint::runtime', u'openstack::barbican::runtime'], u'force': True, u'personalities': [u'controller'], u'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} controller
2020-08-14 03:11:36.536 75323 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/20.06/hieradata'
2020-08-14 03:14:24.214 75323 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'openstack::keystone::endpoint::runtime', u'openstack::barbican::runtime'].
2020-08-14 03:14:30.051 82634 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=28c19440-4bdf-47fe-a583-c79b34b90b6d, classes: ['platform::config::runtime']
2020-08-14 03:14:32.403 82634 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 28c19440-4bdf-47fe-a583-c79b34b90b6d {'classes': ['platform::config::runtime'], 'force': True, 'personalities': ['controller', 'worker', 'storage']} to agent
2020-08-14 03:14:32.408 75323 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 28c19440-4bdf-47fe-a583-c79b34b90b6d {u'classes': [u'platform::config::runtime'], u'force': True, u'personalities': [u'controller', u'worker', u'storage']} controller
2020-08-14 03:14:32.410 75323 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/20.06/hieradata'
2020-08-14 03:14:38.575 75323 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'platform::config::runtime'].
2020-08-14 03:28:16.128 130078 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=36e23f6f-5d80-4ec7-9b5e-75c71caac04e, classes: ['platform::snmp::runtime', 'platform::fm::runtime']
2020-08-14 03:28:19.222 130078 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 36e23f6f-5d80-4ec7-9b5e-75c71caac04e {'classes': ['platform::snmp::runtime', 'platform::fm::runtime'], 'force': False, 'personalities': ['controller']} to agent
2020-08-14 03:28:22.145 130078 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=1bafc0d6-f97f-4035-b01a-5d87ad26e84a, classes: ['platform::haproxy::runtime', 'openstack::keystone::endpoint::runtime', 'openstack::horizon::runtime', 'platform::firewall::runtime']
2020-08-14 03:28:24.950 130078 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 1bafc0d6-f97f-4035-b01a-5d87ad26e84a {'classes': ['platform::haproxy::runtime', 'openstack::keystone::endpoint::runtime', 'openstack::horizon::runtime', 'platform::firewall::runtime'], 'force': False, 'personalities': ['controller']} to agent
2020-08-14 03:28:28.086 130078 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=01c6d03c-1bdd-4b3d-a968-3afe5c4d9baf, classes: ['platform::haproxy::runtime', 'openstack::horizon::runtime']
2020-08-14 03:28:30.777 130078 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 01c6d03c-1bdd-4b3d-a968-3afe5c4d9baf {'classes': ['platform::haproxy::runtime', 'openstack::horizon::runtime'], 'force': False, 'personalities': ['controller']} to agent
2020-08-14 03:28:37.240 130078 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=7992b22e-99df-497f-877d-963bd31fc18d, classes: ['platform::compute::grub::runtime', 'platform::compute::config::runtime']
2020-08-14 03:28:40.122 130078 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 7992b22e-99df-497f-877d-963bd31fc18d {'classes': ['platform::compute::grub::runtime', 'platform::compute::config::runtime'], 'force': True, 'personalities': ['controller', 'worker'], 'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} to agent
2020-08-14 03:28:40.126 75323 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 7992b22e-99df-497f-877d-963bd31fc18d {u'classes': [u'platform::compute::grub::runtime', u'platform::compute::config::runtime'], u'force': True, u'personalities': [u'controller', u'worker'], u'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} controller
2020-08-14 03:28:40.127 75323 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/20.06/hieradata'
2020-08-14 03:28:40.638 130078 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=52e4ecc0-72fb-4864-9ad9-c224d76e9627, classes: ['platform::compute::grub::runtime', 'platform::compute::config::runtime']
2020-08-14 03:28:43.468 130078 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 52e4ecc0-72fb-4864-9ad9-c224d76e9627 {'classes': ['platform::compute::grub::runtime', 'platform::compute::config::runtime'], 'force': True, 'personalities': ['controller', 'worker'], 'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} to agent
2020-08-14 03:28:47.074 130078 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=65334354-e0cc-47ed-814d-0e4c5c67a6b8, classes: ['platform::interfaces::sriov::runtime', 'platform::devices::fpga::fec::runtime']
2020-08-14 03:28:50.366 130078 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 65334354-e0cc-47ed-814d-0e4c5c67a6b8 {'inventory_update': 'pci_sriov_config', 'classes': ['platform::interfaces::sriov::runtime', 'platform::devices::fpga::fec::runtime'], 'force': True, 'personalities': ['controller', 'worker'], 'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} to agent
2020-08-14 03:28:53.211 75323 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'platform::compute::grub::runtime', u'platform::compute::config::runtime'].
2020-08-14 03:28:53.272 75323 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 52e4ecc0-72fb-4864-9ad9-c224d76e9627 {u'classes': [u'platform::compute::grub::runtime', u'platform::compute::config::runtime'], u'force': True, u'personalities': [u'controller', u'worker'], u'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} controller
2020-08-14 03:28:53.273 75323 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/20.06/hieradata'
2020-08-14 03:28:54.110 130078 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=7e48b6ad-883a-4d4a-8a9c-60490357f15c, classes: ['platform::partitions::runtime']
2020-08-14 03:28:59.653 130078 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=5f1d0012-3983-4cd2-832e-d1c15228315e, classes: ['platform::interfaces::sriov::runtime', 'platform::devices::fpga::fec::runtime']
2020-08-14 03:28:59.660 130078 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 7e48b6ad-883a-4d4a-8a9c-60490357f15c {'force': True, 'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12'], 'partition_uuid': u'06789745-cbe7-48b7-b459-5897b2b1484b', 'report_status': 'manage_disk_partitions', 'classes': ['platform::partitions::runtime'], 'idisk_uuid': u'dd19c508-37c5-46be-8a02-84bc8f118504', 'personalities': [u'controller']} to agent
2020-08-14 03:29:02.657 130078 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 5f1d0012-3983-4cd2-832e-d1c15228315e {'inventory_update': 'pci_sriov_config', 'classes': ['platform::interfaces::sriov::runtime', 'platform::devices::fpga::fec::runtime'], 'force': True, 'personalities': ['controller', 'worker'], 'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} to agent
2020-08-14 03:29:06.247 75323 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'platform::compute::grub::runtime', u'platform::compute::config::runtime'].
2020-08-14 03:29:06.289 75323 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 65334354-e0cc-47ed-814d-0e4c5c67a6b8 {u'inventory_update': u'pci_sriov_config', u'classes': [u'platform::interfaces::sriov::runtime', u'platform::devices::fpga::fec::runtime'], u'force': True, u'personalities': [u'controller', u'worker'], u'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} controller
2020-08-14 03:29:06.290 75323 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/20.06/hieradata'
2020-08-14 03:29:13.830 75323 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'platform::interfaces::sriov::runtime', u'platform::devices::fpga::fec::runtime'].
2020-08-14 03:29:54.020 75323 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 7e48b6ad-883a-4d4a-8a9c-60490357f15c {u'force': True, u'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12'], u'partition_uuid': u'06789745-cbe7-48b7-b459-5897b2b1484b', u'idisk_uuid': u'dd19c508-37c5-46be-8a02-84bc8f118504', u'classes': [u'platform::partitions::runtime'], u'report_status': u'manage_disk_partitions', u'personalities': [u'controller']} controller
2020-08-14 03:29:54.021 75323 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/20.06/hieradata'
2020-08-14 03:30:11.991 75323 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'platform::partitions::runtime'].
2020-08-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: TypeError: 'NoneType' object has no attribute '__getitem__'
ck (most recent call last):

"/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data
rgs)

"/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
ult = getattr(proxyobj, method)(ctxt, **kwargs)

"/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 6969, in report_config_status
f.check_pending_app_reapply(context)

"/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 5545, in check_pending_app_reapply
self._check_software_orchestration_in_progress():

"/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 5440, in _check_software_orchestration_in_progress
vim_resp['sw-update-type'] is not None and \

or: 'NoneType' object has no attribute '__getitem__'
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     **args)
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     return Retrying(*dargs, **dkw).call(f, *args, **kw)
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/retrying.py", line 223, in call
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     return attempt.get(self._wrap_exception)
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     six.reraise(self.value[0], self.value[1], self.value[2])
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     return f(*args, **kwargs)
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1651, in config_apply_runtime_manifest
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     error=None)
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1037, in report_config_status
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     error=error))
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 121, in call
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     result = rpc.call(context, real_topic, msg, timeout)
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/__init__.py", line 139, in call
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     return _get_impl().call(CONF, context, topic, msg, timeout)
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/impl_kombu.py", line 799, in call
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     rpc_amqp.get_connection_pool(conf, Connection))
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 620, in call
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     rv = list(rv)
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 569, in __iter__
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     raise result  # pylint: disable=raising-bad-type
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp TypeError: 'NoneType' object has no attribute '__getitem__'
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     **args)
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 6969, in report_config_status
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     self.check_pending_app_reapply(context)
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 5545, in check_pending_app_reapply
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     if self._check_software_orchestration_in_progress():
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 5440, in _check_software_orchestration_in_progress
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp     if vim_resp['sw-update-type'] is not None and \
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp TypeError: 'NoneType' object has no attribute '__getitem__'
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp
-14 03:30:12.050 75323 ERROR sysinv.openstack.common.rpc.amqp
2020-08-14 03:30:12.057 75323 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 5f1d0012-3983-4cd2-832e-d1c15228315e {u'inventory_update': u'pci_sriov_config', u'classes': [u'platform::interfaces::sriov::runtime', u'platform::devices::fpga::fec::runtime'], u'force': True, u'personalities': [u'controller', u'worker'], u'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} controller
2020-08-14 03:30:12.057 75323 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/20.06/hieradata'
2020-08-14 03:30:30.408 75323 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'platform::interfaces::sriov::runtime', u'platform::devices::fpga::fec::runtime'].
2020-08-14 03:30:32.556 130078 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=98997e5c-b3f1-48bf-954c-fa22bebbf719, classes: ['platform::compute::grub::runtime', 'platform::compute::config::runtime']
2020-08-14 03:30:37.953 130078 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 18997e5c-b3f1-48bf-954c-fa22bebbf719 {'classes': ['platform::compute::grub::runtime', 'platform::compute::config::runtime'], 'force': True, 'personalities': ['controller'], 'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} to agent
2020-08-14 03:30:44.560 75323 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 18997e5c-b3f1-48bf-954c-fa22bebbf719 {u'classes': [u'platform::compute::grub::runtime', u'platform::compute::config::runtime'], u'force': True, u'personalities': [u'controller'], u'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} controller
2020-08-14 03:30:44.561 75323 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/20.06/hieradata'
2020-08-14 03:30:57.617 75323 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'platform::compute::grub::runtime', u'platform::compute::config::runtime'].

Boot
2020-08-14T03:34:41.767 controller-0 kernel: notice [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1127.el7.4.tis.x86_64 root=UUID=170af5d4-98ab-4909-b0da-dd83f3da53dc ro module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto console=ttyS0,115200n8 iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1 intel_iommu=on biosdevname=0 user_namespace.enable=1 nopti nospectre_v2 nospectre_v1 hugepagesz=1G hugepages=4 kvm-intel.eptad=0 hugepagesz=2M hugepages=0 default_hugepagesz=1G irqaffinity=0-1 rcu_nocbs=2-43 kthread_cpus=0-1
2020-08-14 03:42:06.220 100560 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=1421686d-b26c-49f0-9718-4ae4f3813086, classes: ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime']
2020-08-14 03:42:11.577 100560 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 1421686d-b26c-49f0-9718-4ae4f3813086 {'classes': ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime'], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} to agent
2020-08-14 03:42:11.586 100560 INFO sysinv.conductor.manager [-] https config applied {'classes': ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime'], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']}
2020-08-14 03:42:16.452 95006 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 1421686d-b26c-49f0-9718-4ae4f3813086 {u'classes': [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime'], u'force': False, u'personalities': [u'controller'], u'host_uuids': [u'535a16f2-525f-44ca-b6d6-27a52165dc12']} controller
2020-08-14 03:42:16.453 95006 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/20.06/hieradata'
2020-08-14 03:43:42.392 95006 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime'].
2020-08-14T03:56:24.000 controller-0 -sh: info HISTORY: PID=225933 UID=42425 system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne host-lock controller-0
2020-08-14T03:57:44.000 controller-0 -sh: info HISTORY: PID=225933 UID=42425 system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne host-unlock controller-0

Boot
2020-08-14T04:00:53.066 controller-0 kernel: notice [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1127.el7.4.tis.x86_64 root=UUID=170af5d4-98ab-4909-b0da-dd83f3da53dc ro module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto console=ttyS0,115200n8 iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1 intel_iommu=on biosdevname=0 user_namespace.enable=1 hugepagesz=1G hugepages=4 kvm-intel.eptad=0 hugepagesz=2M hugepages=0 default_hugepagesz=1G irqaffinity=0-1 rcu_nocbs=2-43 kthread_cpus=0-1 nopti nospectre_v2 nospectre_v1

forced reboot
2020-08-14T04:16:41.000 controller-0 -sh: info HISTORY: PID=41196 UID=42425 sudo reboot -f

Boot
2020-08-14T04:18:42.047 controller-0 kernel: notice [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1127.el7.4.tis.x86_64 root=UUID=170af5d4-98ab-4909-b0da-dd83f3da53dc ro module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto console=ttyS0,115200n8 iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1 intel_iommu=on biosdevname=0 user_namespace.enable=1 hugepagesz=1G hugepages=4 kvm-intel.eptad=0 hugepagesz=2M hugepages=0 default_hugepagesz=1G irqaffinity=0-1 rcu_nocbs=2-43 kthread_cpus=0-1 nopti nospectre_v2 nospectre_v1

Lock
2020-08-14T06:08:56.000 controller-0 -sh: info HISTORY: PID=19111 UID=42425 system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne host-lock controller-0

Unlock
2020-08-14T06:10:05.000 controller-0 -sh: info HISTORY: PID=19111 UID=42425 system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne host-unlock controller-0

Boot
2020-08-14T06:13:17.126 controller-0 kernel: notice [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1127.el7.4.tis.x86_64 root=UUID=170af5d4-98ab-4909-b0da-dd83f3da53dc ro module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto console=ttyS0,115200n8 iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1 intel_iommu=on biosdevname=0 user_namespace.enable=1 hugepagesz=1G hugepages=4 kvm-intel.eptad=0 hugepagesz=2M hugepages=0 default_hugepagesz=1G irqaffinity=0-1 rcu_nocbs=2-43 kthread_cpus=0-1 nopti nospectre_v2 nospectre_v1

Runtime manifests
2020-08-14 06:48:09.377 88144 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=086cf255-3782-45df-98d8-1ba8e7e16189, classes: ['platform::collectd::restart']
2020-08-14 06:48:13.876 88144 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 086cf255-3782-45df-98d8-1ba8e7e16189 {'classes': ['platform::collectd::restart'], 'force': False, 'personalities': ['controller', 'worker']} to agent
2020-08-14 06:48:13.880 3535 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 086cf255-3782-45df-98d8-1ba8e7e16189 {u'classes': [u'platform::collectd::restart'], u'force': False, u'personalities': [u'controller', u'worker']} controller
2020-08-14 06:48:13.881 3535 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/20.06/hieradata'
2020-08-14 06:48:23.641 3535 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'platform::collectd::restart'].
2020-08-14 06:52:42.922 88144 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=1f37c378-cbc2-4dad-a345-667384ebb637, classes: ['platform::collectd::restart']
2020-08-14 06:52:47.110 88144 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 1f37c378-cbc2-4dad-a345-667384ebb637 {'classes': ['platform::collectd::restart'], 'force': False, 'personalities': ['controller', 'worker']} to agent
2020-08-14 06:52:47.115 3535 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 1f37c378-cbc2-4dad-a345-667384ebb637 {u'classes': [u'platform::collectd::restart'], u'force': False, u'personalities': [u'controller', u'worker']} controller
2020-08-14 06:52:47.116 3535 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/20.06/hieradata'
2020-08-14 06:53:01.637 3535 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'platform::collectd::restart'].

host lock/unlock
2020-08-14T07:03:49.000 controller-0 -sh: info HISTORY: PID=35460 UID=42425 system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne host-lock controller-0
2020-08-14T07:05:08.000 controller-0 -sh: info HISTORY: PID=35460 UID=42425 system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne host-unlock controller-0

boot
2020-08-14T07:08:20.144 controller-0 kernel: notice [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1127.el7.4.tis.x86_64 root=UUID=170af5d4-98ab-4909-b0da-dd83f3da53dc ro module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto console=ttyS0,115200n8 iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1 intel_iommu=on biosdevname=0 user_namespace.enable=1 hugepagesz=1G hugepages=4 kvm-intel.eptad=0 hugepagesz=2M hugepages=0 default_hugepagesz=1G irqaffinity=0-1 rcu_nocbs=2-43 kthread_cpus=0-1 nopti nospectre_v2 nospectre_v1

It looks like mtcAgent (port 2112) is not started, when
nfvi makes api requests for host list, failes and
it's looks like the reason for the alarm with configuration failure.

2020-08-14T04:38:50.474 controller-0                           VIM_Thread[104681]    ERROR Caught exception while trying to get host list, error=[OpenStack Exception: method=GET, url=http://localhost:2112/v1/hosts/535a16f2-525f-44ca-b6d6-27a52165dc12, headers={'Content-Type': 'application/json', 'User-Agent': 'vim/1.0'}, body={"hostname": "controller-0", "uuid": "535a16f2-525f-44ca-b6d6-27a52165dc12"}, reason=<urlopen error [Errno 111] Connection refused>].
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nfv_plugins/nfvi_plugins/nfvi_infrastructure_api.py", line 395, in get_hosts
    future.result = (yield)
OpenStackException: [OpenStack Exception: method=GET, url=http://localhost:2112/v1/hosts/535a16f2-525f-44ca-b6d6-27a52165dc12, headers={'Content-Type': 'application/json', 'User-Agent': 'vim/1.0'}, body={"hostname": "controller-0", "uuid": "535a16f2-525f-44ca-b6d6-27a52165dc12"}, reason=<urlopen error [Errno 111] Connection refused>]
2020-08-14T04:38:50.475 controller-0                           VIM_Thread[104681]    ERROR                        _vim_nfvi_audits.py.187   Audit-Hosts callback, not completed, responses={'completed': False, 'host-groups': [], 'reason': '', 'incomplete-hosts': []}.
2020-08-14T04:38:58.751 controller-0                           VIM_Thread[104681]    ERROR Caught exception while trying to get host list, error=[OpenStack Exception: method=GET, url=http://localhost:2112/v1/hosts/535a16f2-525f-44ca-b6d6-27a52165dc12, headers={'Content-Type': 'application/json', 'User-Agent': 'vim/1.0'}, body={"hostname": "controller-0", "uuid": "535a16f2-525f-44ca-b6d6-27a52165dc12"}, reason=<urlopen error [Errno 111] Connection refused>].
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nfv_plugins/nfvi_plugins/nfvi_infrastructure_api.py", line 395, in get_hosts
    future.result = (yield)
OpenStackException: [OpenStack Exception: method=GET, url=http://localhost:2112/v1/hosts/535a16f2-525f-44ca-b6d6-27a52165dc12, headers={'Content-Type': 'application/json', 'User-Agent': 'vim/1.0'}, body={"hostname": "controller-0", "uuid": "535a16f2-525f-44ca-b6d6-27a52165dc12"}, reason=<urlopen error [Errno 111] Connection refused>]
2020-08-14T04:38:58.751 controller-0                           VIM_Thread[104681]    ERROR                        _vim_nfvi_audits.py.187   Audit-Hosts callback, not completed, responses={'completed': False, 'host-groups': [], 'reason': '', 'incomplete-hosts': []}.

Revision history for this message

Paul-Ionut Vaduva (pvaduva) wrote on 2020-12-11:

The second issue on the DX is due to a
PUppet manifest failed with tiller failed to restart

var/log/puppet/2020-08-28-03-39-41_controller/puppet.log

-----
2020-08-28T03:40:46.947 Notice: 2020-08-28 03:40:46 +0000 /Stage[main]/Platform::Helm::Tiller::Config/Exec[restart tiller for helm]/returns: Job for tiller.service failed because the control process exited with error c│
│2020-08-28T03:40:46.949 Error: 2020-08-28 03:40:46 +0000 /Stage[main]/Platform::Helm::Tiller::Config/Exec[restart tiller for helm]: Failed to call refresh: systemctl restart tiller.service returned 1 instead of one of │
│2020-08-28T03:40:46.952 Error: 2020-08-28 03:40:46 +0000 /Stage[main]/Platform::Helm::Tiller::Config/Exec[restart tiller for helm]: systemctl restart tiller.service returned 1 instead of one of [0] │
│2020-08-28T03:40:46.954 /usr/share/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail' │------

Revision history for this message

Paul-Ionut Vaduva (pvaduva) wrote on 2020-12-23:

We did not see this in the last 1-2 months so, I suggest to open an other issue if we see it in the future.

Changed in starlingx:
status:	Triaged → Invalid

Ghada Khalil (gkhalil) on 2021-10-27

tags:

removed: stx.retestneeded

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.