OVB jobs are failing introspection in RDOcloud - bmc-template login errors

[Ronelle Landy]

From 11/07, there are consistent failures in introspection on all OVB tests in rdocloud and in accessing the supplemental node in fs039 tests.

The tests running in vexxhost are passing.

See the following results comparison:

https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001

vs

https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost

The introspection failure logs show that we have no IPMI access:

http://logs.rdoproject.org/openstack-regular/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035/81df48d/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz

IPMI Error while attempting "ipmitool -I lanplus -H 192.168.101.154 -L ADMINISTRATOR -U admin -v -R 12 -N 5 -f /tmp/tmpeF3z_6 power status" for node f6bfe30f-3df9-486b-b3da-f13b753daac4. Error: Unexpected error while running command.

Command: ipmitool -I lanplus -H 192.168.101.166 -L ADMINISTRATOR -U admin -v -R 12 -N 5 -f /tmp/tmpOuZMaJ power status
Exit code: 1
Stdout: u''
Stderr: u'Get Auth Capabilities error\nError issuing Get Channel Authentication Capabilities request\nError: Unable to establish IPMI v2 / RMCP+ session\n': ProcessExecutionError: Unexpected error while running command.

The bmc log shows:
bmc-template login: [ 96.339876] cloud-init[779]: 2019-11-07 22:21:55,664 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
http://logs.rdoproject.org/openstack-regular/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035/81df48d/logs/bmc-console.log

The stack creation shows:

2019-11-07 22:15:47.390667 | TASK [ovb-manage : Print ovb args]
2019-11-07 22:15:47.470067 | OVB args:'--name baremetal_914 --quintupleo --id 914 -e /home/zuul/workspace/ovb/env-914-base.yaml -e /home/zuul/workspace/ovb/openstack-virtual-baremetal/environments/bmc-use-cache.yaml -e /home/zuul/workspace/ovb/openstack-virtual-baremetal/environments/undercloud-floating-none.yaml -e /home/zuul/workspace/ovb/openstack-virtual-baremetal/environments/quintupleo-no-undercloud.yaml -e /home/zuul/workspace/ovb/openstack-virtual-baremetal/environments/all-networks-port-security.yaml '
2019-11-07 22:15:47.501392 |
2019-11-07 22:15:47.501635 | TASK [ovb-manage : Find out UUID of instance with metadata URL]
2019-11-07 22:17:55.243857 | primary | Traceback (most recent call last):
2019-11-07 22:17:55.243992 | primary | File "<string>", line 1, in <module>
2019-11-07 22:17:55.244167 | primary | File "/usr/lib64/python2.7/json/__init__.py", line 290, in load
2019-11-07 22:17:55.245418 | primary | **kw)
2019-11-07 22:17:55.245485 | primary | File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
2019-11-07 22:17:55.245519 | primary | return _default_decoder.decode(s)
2019-11-07 22:17:55.245565 | primary | File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
2019-11-07 22:17:55.246791 | primary | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2019-11-07 22:17:55.246871 | primary | File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
2019-11-07 22:17:55.246921 | primary | raise ValueError("No JSON object could be decoded")
2019-11-07 22:17:55.246988 | primary | ValueError: No JSON object could be decoded
2019-11-07 22:17:55.718011 | primary | ERROR
2019-11-07 22:17:55.718373 | primary | {
2019-11-07 22:17:55.718500 | primary | "delta": "0:02:07.249268",
2019-11-07 22:17:55.718572 | primary | "end": "2019-11-07 22:17:55.248061",
2019-11-07 22:17:55.718647 | primary | "msg": "non-zero return code",
2019-11-07 22:17:55.718713 | primary | "rc": 1,
2019-11-07 22:17:55.718771 | primary | "start": "2019-11-07 22:15:47.998793"
2019-11-07 22:17:55.718829 | primary | }
2019-11-07 22:17:55.718903 | primary | ERROR: Ignoring Errors

One possibility is the recent changes in openstack/python-novaclient.

[Ronelle Landy (rlandy)]

kforde fix this ... "As Yatin points out the prerouting rule bug that hit us before reappeared.
I've restored the rule and things should be working now."

Closing this bug out