network validation tests fail: No module named ipaddr

Bug #1551048 reported by Dan Prince
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Dan Prince

Bug Description

The all nodes network validation in t-h-t fails to run correctly. It fails in such a way that it doesn't actually cause a Heat software deployment failure so the failure remains silent. The following error message is logged to os-collect-config.log:

Feb 29 00:53:55 overcloud-controller-0 os-collect-config[5346]: [2016-02-29 00:53:55,530] (heat-config) [INFO] {"deploy_stdout": "Trying to ping default gateway 10.0.0.1...SUCCESS\n", "deploy_stderr": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\nTraceback (most recent call last):\n File \"<string>\"

----

The root cause of the failure seems to be this IPv6 commit where we switch to using python which imports this library:

commit 45be848991eb2da0ed039dcaa457800b8ed3f314

    Adds v6 capability to the deploy validation test (pings)

Dan Prince (dan-prince)
Changed in tripleo:
assignee: nobody → Dan Prince (dan-prince)
importance: Undecided → Critical
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/285857

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-puppet-elements (master)

Fix proposed to branch: master
Review: https://review.openstack.org/285858

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-puppet-elements (master)

Reviewed: https://review.openstack.org/285858
Committed: https://git.openstack.org/cgit/openstack/tripleo-puppet-elements/commit/?id=b217175340dfb10bdbaf158c00ba2535afd4bde9
Submitter: Jenkins
Branch: master

commit b217175340dfb10bdbaf158c00ba2535afd4bde9
Author: Dan Prince <email address hidden>
Date: Sun Feb 28 20:28:03 2016 -0500

    Add overcloud-base element

    This patch adds a new overcloud-base element that currently
    installs the python-ipaddr package. Per t-h-t commit 45be848
    python-ipaddr is now required for network validations on
    all nodes.

    Change-Id: Ia617f44b4673b89202e5e5cdcac9b50f46b3e6c8
    Closes-bug: #1551048

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-puppet-elements (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/286553

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-puppet-elements (stable/liberty)

Reviewed: https://review.openstack.org/286553
Committed: https://git.openstack.org/cgit/openstack/tripleo-puppet-elements/commit/?id=7bb470334ac9777cce6537d98a0247cdddf1a20b
Submitter: Jenkins
Branch: stable/liberty

commit 7bb470334ac9777cce6537d98a0247cdddf1a20b
Author: Dan Prince <email address hidden>
Date: Sun Feb 28 20:28:03 2016 -0500

    Add overcloud-base element

    This patch adds a new overcloud-base element that currently
    installs the python-ipaddr package. Per t-h-t commit 45be848
    python-ipaddr is now required for network validations on
    all nodes.

    Change-Id: Ia617f44b4673b89202e5e5cdcac9b50f46b3e6c8
    Closes-bug: #1551048
    (cherry picked from commit b217175340dfb10bdbaf158c00ba2535afd4bde9)

tags: added: in-stable-liberty
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/285857
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=bbb38534866c96e7688618b41449cae07871f3a2
Submitter: Jenkins
Branch: master

commit bbb38534866c96e7688618b41449cae07871f3a2
Author: Dan Prince <email address hidden>
Date: Sun Feb 28 20:19:49 2016 -0500

    Use set -e for validation-scripts/all-nodes.sh

    Update this script to use 'set -e' for all commands except
    the ping checks themselves which we allow to fail so we
    can give enhanced output.

    This resolves an issue where some commands could fail and cause
    an undetectable error (the scripted exits with success) thus causing
    a case where Heat won't detect any network errors at all. This
    was recently the case with git commit 45be848 where we switched
    to use python -c 'import ipaddr' which wasn't even installed as a
    library on our overcloud nodes, thus causing all network validations
    to silently fail.

    Change-Id: I40ed6a537136e866357cc0d9304e905afdd28522
    Depends-On: Ia617f44b4673b89202e5e5cdcac9b50f46b3e6c8
    Related-bug: #1551048

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.