legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 and 35 failing overcloud_prep_images with 'Connection is already closed

[Marios Andreou]

The legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master /queens/pike and legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master are failing with

 2018-09-28 01:17:44 | Introspection completed.
 2018-09-28 01:17:44 | + openstack overcloud node provide --all-manageable
 2018-09-28 01:17:48 | Waiting for messages on queue 'tripleo' with no timeout.
 2018-09-28 02:19:42 | Connection is already closed.
 2018-09-28 02:19:42 | [{}, {u'result': u'Node 5414a384-c195-41e6-838c-fec37d6247dc did not reach state "available", the state is "clean failed", error: Failed to prepare node 5414a384-c195-41e6-838c-fec37d6247dc for cleaning: Failed to set node power state to power on.'}, {}, {u'result': u'Node 431d2602-e007-425f-987e-361e7900935f did not reach state "available", the state is "clean failed", error: Failed to prepare node 431d2602-e007-425f-987e-361e7900935f for cleaning: Failed to set node power state to power on.'}]

Examples: fs 001: master [1,2,3] queens [4,5], pike [6] but on the 26th
          fs 035: master [7]

[1] http://logs.rdoproject.org/58/599358/4/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/5451077/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz
[2] http://logs.rdoproject.org/80/604180/16/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/11ccd7b/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz
[3] http://logs.rdoproject.org/78/605678/2/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/c4c128d/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz
[4] http://logs.rdoproject.org/00/604800/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/70b467e/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz
[5] http://logs.rdoproject.org/91/603391/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/465f4cb/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz
[6] http://logs.rdoproject.org/14/597814/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike/79afd76/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz
[7] http://logs.rdoproject.org/49/600849/24/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/1d7b689/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz

[wes hayutin (weshayutin)]

Adding an alert as there are only 2 other alert bugs atm, and this is causing inconsistent 3rd party tripleo results.

[Marios Andreou (marios-b)]

I think this is a duplicate of https://bugs.launchpad.net/tripleo/+bug/1792872 - the hesitation is because the traces are slightly different in the various examples (see description above and comment #3 at bug/1792872 for examples) however in all cases the error is happening during the Ironic introspection during the overcloud-prep-images.sh and is coming from mistral. Will add more info in the other bug for now thanks.