ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq

[Marios Andreou]

The ping test is sometimes failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq job. The stack is at create complete but the pingtest stack creation fails. I came across this trying to land this https://review.openstack.org/#/c/504038 - pointers to logs:

    http://logs.openstack.org/38/504038/3/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/d255edd/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-09-19_08_03_11
    http://logs.openstack.org/38/504038/3/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/d255edd/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-09-19_07_58_59

Went looking and found a couple more examples here:

    https://review.openstack.org/#/c/498898/3
    http://logs.openstack.org/98/498898/3/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/586f327/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-09-19_14_45_43

    https://review.openstack.org/#/c/505153/2
    http://logs.openstack.org/53/505153/2/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/6fdbb36/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-09-19_12_08_48

[Marios Andreou (marios-b)]

just posted an elastic recheck query for this one so we can see how frequent it is https://review.openstack.org/505574

[Jose Luis Franco (jfrancoa)]

There is a traceback which could give some hint in heat-engine:
http://logs.openstack.org/38/504038/3/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/d255edd/logs/subnode-2/var/log/heat/heat-engine.log.txt.gz#_2017-09-19_08_03_25_928

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Hi Marios,

another one:

http://logs.openstack.org/67/474967/28/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/ecb43a8/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-09-28_12_35_09

I've added a bit of debug log from there https://review.openstack.org/#/q/I18a73b2936975c056ebc3f4b984349b61a4ed58d on this review https://review.openstack.org/#/c/474967/

Waiting for the output....

[Jose Luis Franco (jfrancoa)]

Checking some more logs, the problem seems to come from nova:

2017-09-28 12:31:16.872 66756 ERROR nova.compute.manager [instance: ba7c96d7-1179-4c52-bf12-8a6b1d94d190] NovaException: Unable to get host UUID: /etc/machine-id is empty
2017-09-28 12:31:16.872 66756 ERROR nova.compute.manager [instance: ba7c96d7-1179-4c52-bf12-8a6b1d94d190]

http://logs.openstack.org/67/474967/28/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/ecb43a8/logs/subnode-2/var/log/nova/nova-compute.log.txt.gz#_2017-09-28_12_31_16_872

There is a related bugzila, although it was affecting the containerized deployment: https://bugzilla.redhat.com/show_bug.cgi?id=1464182 [CLOSED]

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Hi,

thanks to bandini's comment about heat engine log, here is the error:

http://logs.openstack.org/67/474967/28/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/ecb43a8/logs/subnode-2/var/log/heat/heat-engine.log.txt.gz#_2017-09-28_12_31_28_313

2017-09-28 12:31:28.313 71973 ERROR heat.engine.resource ResourceInError: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2017-09-28 12:31:28.313 71973 ERROR heat.engine.resource

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

The final error is that:

http://logs.openstack.org/67/474967/28/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/ecb43a8/logs/subnode-2/var/log/nova/nova-conductor.log.txt.gz#_2017-09-28_12_34_39_463

File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1981, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 6d97341e-fdaa-4c95-9b7b-bbbc16edcafd was re-scheduled: Unable to get host UUID: /etc/machine-id is empty\n']

subnode-2 doesn't have /etc/machine-id

we need to have

 [ -e /etc/machine-id ] || \
       systemd-machine-id-setup

run somewhere.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Oki, my bad. Previous comment was wrong as the machine-id is properly setup during bootup:

Sep 28 10:52:17 localhost systemd[1]: Initializing machine ID from random generator.[1]

but it's deleted by instack:

Sep 28 08:31:20 centos-7-2-node-rax-iad-11184996-929537 sudo: jenkins : TTY=unknown ; PWD=/home/jenkins ; USER=root ; COMMAND=/bin/instack -e centos7 enable-packages-install install-types selinux-permissive hosts baremetal dhcp-all-interfaces os-collect-config overcloud-full overcloud-controller overcloud-compute overcloud-ceph-storage puppet-modules hiera os-net-config stable-interface-names grub2 element-manifest network-gateway dynamic-login enable-packages-install pip-and-virtualenv-override remove-machine-id -k extra-data pre-install install post-install -b 05-fstab-rootfs-label 00-fix-requiretty 90-rebuild-ramdisk 00-usr-local-bin-secure-path -x delorean-repo -d

Look for the remove-machine-id element from https://github.com/openstack/tripleo-puppet-elements

[1]: http://logs.openstack.org/67/474967/28/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/ecb43a8/logs/subnode-2/var/log/messages.txt.gz#_Sep_28_10_52_17

[2]: http://logs.openstack.org/67/474967/28/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/5b3a972/logs/subnode-2/var/log/secure.txt.gz#_Sep_28_08_31_20

[Alex Schultz (alex-schultz)]

remove-machine-id is needed for image building but if run on an existing system it would be a problem. It seems that needs to be excluded from CI for this

[Alex Schultz (alex-schultz)]

remove-machine-id needs to be excluded in:
https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/bootstrap-overcloud-full.sh#L53-L70

[Michele Baldessari (michele)]

https://review.openstack.org/#/c/508226/ should fix this, once merged

[Michele Baldessari (michele)]

https://review.openstack.org/#/c/508226/ has merged. Removing alert.

[Emilien Macchi (emilienm)]

re-opening it, it fails on newton (today): http://logs.openstack.org/18/510218/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/6b87f98/logs/subnode-2/var/log/nova/nova-compute.log.txt.gz#_2017-10-06_22_19_46_429

[Emilien Macchi (emilienm)]

Please review https://review.openstack.org/510261

[Emilien Macchi (emilienm)]

Fixed in https://review.openstack.org/#/c/510312/ in fact. Please review