Multinode jobs fail with "get_image_element_array: command not found"

[Ben Nemec]

We're seeing a lot of multinode jobs failing on this error. It's not clear to me if they _all_ are because I see a lot more failures in logstash than on the cistatus page, which confuses me.

Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%20*tripleo-ci*%20AND%20build_status%3A%20FAILURE%20AND%20message%3A%20%5C%22get_image_element_array%3A%20command%20not%20found%5C%22

Last I checked we were at 110 failed jobs, and rising steadily.

[Ben Nemec (bnemec)]

One interesting thing I noticed is that a job which passed was getting the dib elements from /usr/share/diskimage-builder/elements and the failing ones from /usr/lib/python2.7/site-packages/diskimage_builder/elements

We noticed that instack has a hard-coded reference to the former, but I don't think it should ever be getting to that code path: https://github.com/openstack/instack/blob/master/instack/main.py#L105

I checked locally and dib is correctly returning the /usr/lib path from its get_path call so instack should use that.

[Ben Nemec (bnemec)]

https://review.openstack.org/#/c/425888/ may work around this by not using instack in multinode. It seems dib 2.0 was merged into master today and it has some changes that broke the way multinode uses instack.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/443367

[James Slagle (james-slagle)]

either https://review.openstack.org/#/c/425888/ or https://review.openstack.org/443367 should fix this issue.

the former circumvents the issue by not using instack to bootstrap subnodes, something that we need to land eventually.

the latter is a fix in instack needed for the dib v2 changes

[James Slagle (james-slagle)]

https://review.openstack.org/#/c/425888/ is merged at least, so removing alert tag

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/443367
Committed: https://git.openstack.org/cgit/openstack/instack/commit/?id=db7123ec103c192557cbf64656ba4be2499c94da
Submitter: Jenkins
Branch: master

commit db7123ec103c192557cbf64656ba4be2499c94da
Author: James Slagle <email address hidden>
Date: Wed Mar 8 19:01:57 2017 -0500

    Generate and source a dib v2 environment

    diskimage-builder v2 now relies on the output of "element-info --env
    <elements>" to have been sourced into the shell running the element
    hooks. Generate the environment and save the output to a file under
    environment.d so that dib-run-parts will source it prior to running
    hooks.

    Change-Id: Ie8ac128e391a7091184b5017959a2a653166062f
    Closes-Bug: #1671262

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/instack 7.0.0 release.