Introspection fails on OVB jobs on master and ussuri.
Example from vexxhost:
In vexxhost there is error in BMC host:
https://logserver.rdoproject.org/28/735428/4/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-vexxhost/5a864cc/logs/bmc_4_26064-console.log
Traceback (most recent call last):
[17870.604483] openstackbmc[26746]: File "/usr/local/bin/openstackbmc", line 335, in <module>
[17870.608469] openstackbmc[26746]: main()
[17870.610479] openstackbmc[26746]: File "/usr/local/bin/openstackbmc", line 331, in main
[17870.614361] openstackbmc[26746]: mybmc.listen()
[17870.616601] openstackbmc[26746]: File "/usr/lib/python2.7/site-packages/pyghmi/ipmi/bmc.py", line 153, in listen
[17870.620767] openstackbmc[26746]: ipmisession.Session.wait_for_rsp(timeout)
[17870.624028] openstackbmc[26746]: File "/usr/lib/python2.7/site-packages/pyghmi/ipmi/private/session.py", line 1103, in wait_for_rsp
[17870.628784] openstackbmc[26746]: relsession.process_pktqueue()
[17870.631241] openstackbmc[26746]: File "/usr/lib/python2.7/site-packages/pyghmi/ipmi/private/serversession.py", line 290, in process_pktqueue
[17870.636857] openstackbmc[26746]: self.sessionless_data(pkt[0], pkt[1])
[17870.639367] openstackbmc[26746]: File "/usr/lib/python2.7/site-packages/pyghmi/ipmi/private/serversession.py", line 316, in sessionless_data
[17870.643599] openstackbmc[26746]: myaddr, netfnlun = struct.unpack('2B', bytes(data[14:16]))
[17870.646737] openstackbmc[26746]: struct.error: unpack requires a string argument of length 2
Ironic failures: https://logserver.rdoproject.org/28/735428/4/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-vexxhost/5a864cc/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz
2020-06-17 09:46:37.522 7 WARNING ironic.drivers.modules.ipmitool [req-b0e48544-0133-496b-a59f-125435f89200 - - - - -] IPMI Error encountered, retrying "ipmitool -I lanplus -H 192.168.101.154 -L ADMINISTRATOR -U admin -R 1 -N 1 -f /tmp/tmpfonl9rmo power status" for node 5130f8a7-64b1-4d38-99c4-c76e36c79ba5. Error: Unexpected error while running command.
Command: ipmitool -I lanplus -H 192.168.101.154 -L ADMINISTRATOR -U admin -R 1 -N 1 -f /tmp/tmpfonl9rmo power status
Exit code: 1
Stdout: ''
Stderr: 'Unable to Get Channel Cipher Suites\nError: Received an Unexpected Open Session Response\nError: Unable to establish IPMI v2 / RMCP+ session\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
On RDO cloud BMC looks OK: https://logserver.rdoproject.org/10/735810/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/0293b60/logs/bmc_1_21919-console.log
Seen only a few errors: https://logserver.rdoproject.org/10/735810/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/0293b60/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz
2020-06-17 11:04:58.451 7 WARNING ironic.drivers.modules.ipmitool [req-94360948-7da6-48d8-89f5-31124168bbe1 - - - - -] IPMI Error encountered, retrying "ipmitool -I lanplus -H 192.168.100.184 -L ADMINISTRATOR -U admin -R 1 -N 1 -f /tmp/tmp8ayrxbxg power status" for node 76339813-4fb9-4df4-94b0-6d9d27ba3fb6. Error: Unexpected error while running command.
Command: ipmitool -I lanplus -H 192.168.100.184 -L ADMINISTRATOR -U admin -R 1 -N 1 -f /tmp/tmp8ayrxbxg power status
Exit code: 1
Stdout: ''
Stderr: 'Unable to Get Channel Cipher Suites\nUnable to get Chassis Power Status\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
But still introspection fails: https://logserver.rdoproject.org/10/735810/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/0293b60/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz
2020-06-17 07:54:41 | 2020-06-17 07:54:41.893375 | fa163e6e-e390-580b-6896-00000000000c | FATAL | Notice | localhost | error={"changed": false, "msg": "No nodes are manageable at this time."}
BMC on rdo cloud:
Centos 7, python 2.7.5
pyghmi==1.2.16
BMC on vexxhost:
Centos 7, python 2.7.5
pyghmi==1.0.18