Multinode jobs fail with "get_image_element_array: command not found"

Bug #1671262 reported by Ben Nemec on 2017-03-08
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
James Slagle

Bug Description

We're seeing a lot of multinode jobs failing on this error. It's not clear to me if they _all_ are because I see a lot more failures in logstash than on the cistatus page, which confuses me.

Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%20*tripleo-ci*%20AND%20build_status%3A%20FAILURE%20AND%20message%3A%20%5C%22get_image_element_array%3A%20command%20not%20found%5C%22

Last I checked we were at 110 failed jobs, and rising steadily.

Tags: ci Edit Tag help
Ben Nemec (bnemec) wrote :

One interesting thing I noticed is that a job which passed was getting the dib elements from /usr/share/diskimage-builder/elements and the failing ones from /usr/lib/python2.7/site-packages/diskimage_builder/elements

We noticed that instack has a hard-coded reference to the former, but I don't think it should ever be getting to that code path: https://github.com/openstack/instack/blob/master/instack/main.py#L105

I checked locally and dib is correctly returning the /usr/lib path from its get_path call so instack should use that.

Ben Nemec (bnemec) wrote :

https://review.openstack.org/#/c/425888/ may work around this by not using instack in multinode. It seems dib 2.0 was merged into master today and it has some changes that broke the way multinode uses instack.

Fix proposed to branch: master
Review: https://review.openstack.org/443367

Changed in tripleo:
assignee: nobody → James Slagle (james-slagle)
status: Triaged → In Progress
James Slagle (james-slagle) wrote :

either https://review.openstack.org/#/c/425888/ or https://review.openstack.org/443367 should fix this issue.

the former circumvents the issue by not using instack to bootstrap subnodes, something that we need to land eventually.

the latter is a fix in instack needed for the dib v2 changes

James Slagle (james-slagle) wrote :

https://review.openstack.org/#/c/425888/ is merged at least, so removing alert tag

tags: removed: alert

Reviewed: https://review.openstack.org/443367
Committed: https://git.openstack.org/cgit/openstack/instack/commit/?id=db7123ec103c192557cbf64656ba4be2499c94da
Submitter: Jenkins
Branch: master

commit db7123ec103c192557cbf64656ba4be2499c94da
Author: James Slagle <email address hidden>
Date: Wed Mar 8 19:01:57 2017 -0500

    Generate and source a dib v2 environment

    diskimage-builder v2 now relies on the output of "element-info --env
    <elements>" to have been sourced into the shell running the element
    hooks. Generate the environment and save the output to a file under
    environment.d so that dib-run-parts will source it prior to running
    hooks.

    Change-Id: Ie8ac128e391a7091184b5017959a2a653166062f
    Closes-Bug: #1671262

Changed in tripleo:
status: In Progress → Fix Released

This issue was fixed in the openstack/instack 7.0.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers