The stuck alarm occurs near this log
2019-08-27 08:15:31.123 95305 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 3ac0c536-57ad-4e49-ba37-6877ef030622 {'classes': ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime', 'platform::sysinv::runtime'], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'94146b1f-8449-4cd8-888b-0cffa07293f8']} to agent 2019-08-27 08:15:31.132 95305 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.2:5672 2019-08-27 08:15:31.223 95305 INFO sysinv.conductor.manager [-] drbd-overview: pgsql-40.0, cgcs-20.0, extension-0.96875, patch-vault-0, etcd-4.8, dockerdistribution-16.0 2019-08-27 08:15:31.223 95305 INFO sysinv.conductor.manager [-] lvdisplay: pgsql-40.0, cgcs-20.0, extension-1.0, patch-vault-0, etcd-5.0, dockerdistribution-16.0 2019-08-27 08:15:31.235 95305 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config 328f0cd7-a654-4419-8320-18f619dc2ea4 2019-08-27 08:16:03.390 14322 INFO sysinv.agent.manager [-] get_ihost_by_macs rpc Timeout. 2019-08-27 08:16:03.399 14322 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.2:5672 2019-08-27 08:16:03.417 14322 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 1.728777 sec
We can determine what area of the code is being run, but not why there is no puppet log for that time period.
The stuck alarm occurs near this log
2019-08-27 08:15:31.123 95305 INFO sysinv.agent.rpcapi [-] config_ apply_runtime_ manifest: fanout_cast: sending config 3ac0c536- 57ad-4e49- ba37-6877ef0306 22 {'classes': ['openstack: :keystone: :endpoint: :runtime' , 'platform: :firewall: :runtime' , 'platform: :sysinv: :runtime' ], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'94146b1f- 8449-4cd8- 888b-0cffa07293 f8']} to agent openstack. common. rpc.common [-] Connected to AMQP server on 192.168.204.2:5672 conductor. manager [-] drbd-overview: pgsql-40.0, cgcs-20.0, extension-0.96875, patch-vault-0, etcd-4.8, dockerdistribut ion-16. 0 conductor. manager [-] lvdisplay: pgsql-40.0, cgcs-20.0, extension-1.0, patch-vault-0, etcd-5.0, dockerdistribut ion-16. 0 conductor. manager [-] SYS_I Clear system config alarm: controller-0 target config 328f0cd7- a654-4419- 8320-18f619dc2e a4 agent.manager [-] get_ihost_by_macs rpc Timeout. openstack. common. rpc.common [-] Connected to AMQP server on 192.168.204.2:5672 openstack. common. loopingcall [-] task run outlasted interval by 1.728777 sec
2019-08-27 08:15:31.132 95305 INFO sysinv.
2019-08-27 08:15:31.223 95305 INFO sysinv.
2019-08-27 08:15:31.223 95305 INFO sysinv.
2019-08-27 08:15:31.235 95305 INFO sysinv.
2019-08-27 08:16:03.390 14322 INFO sysinv.
2019-08-27 08:16:03.399 14322 INFO sysinv.
2019-08-27 08:16:03.417 14322 WARNING sysinv.
We can determine what area of the code is being run, but not why there is no puppet log for that time period.