Inspector grenade fails in test_network_basic_ops

Bug #1621791 reported by Dmitry Tantsur
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic Inspector
Fix Released
Medium
milan k

Bug Description

Roughly on Sep 2 it started failing with SSH timeout while checking external connectivity in roughly 90% cases (sometimes it does pass). Example: http://logs.openstack.org/62/365662/6/gate/gate-grenade-dsvm-ironic-inspector/a3398ff/logs/grenade.sh.txt.gz#_2016-09-08_08_51_23_532

UPD1: it seems like we got broken by this Tempest change: https://review.openstack.org/#/c/358610/. Reverting it brings us back to normal operation (as proved by https://review.openstack.org/#/c/366602/), but apparently breaks someone else.

UPD2: we've disabled the failing test, but it will probably be good to re-enable it eventually.

UPD3: the way we've disabled the test disables it for everyone who has our plugin installed, which is not good. However, fixing it in our bash scripts is not an easy task...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic-inspector (master)

Fix proposed to branch: master
Review: https://review.openstack.org/367841

Changed in ironic-inspector:
assignee: nobody → Dmitry Tantsur (divius)
status: Triaged → In Progress
Dmitry Tantsur (divius)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic-inspector (master)

Reviewed: https://review.openstack.org/367841
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=16661685aa7ecb48bce175faeb47c4c919c324b8
Submitter: Jenkins
Branch: master

commit 16661685aa7ecb48bce175faeb47c4c919c324b8
Author: Dmitry Tantsur <email address hidden>
Date: Fri Sep 9 11:03:27 2016 +0200

    Disable neutron tests in our grenade

    We're suffering from unclear failures test_network_basic_ops.
    As these tests are not important for our service, disabling them for now
    by pretending that Neutron is not available.

    Also fix incorrect list of required services on our smoke test, it only
    requires Ironic and Swift.

    Change-Id: Ia0f0976c2516e853482277a1a1045b4a951dec7c
    Partial-Bug: #1621791

Revision history for this message
Dmitry Tantsur (divius) wrote :

We have a work around now, lowering the priority.

Changed in ironic-inspector:
importance: Critical → Medium
Dmitry Tantsur (divius)
description: updated
Changed in ironic-inspector:
status: In Progress → Confirmed
assignee: Dmitry Tantsur (divius) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic-inspector (master)

Fix proposed to branch: master
Review: https://review.openstack.org/387403

Changed in ironic-inspector:
assignee: nobody → Dmitry Tantsur (divius)
status: Confirmed → In Progress
Dmitry Tantsur (divius)
description: updated
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic-inspector (master)

Reviewed: https://review.openstack.org/387403
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=4eb77865ab140c7a10e33c0a6fed201e39bac183
Submitter: Jenkins
Branch: master

commit 4eb77865ab140c7a10e33c0a6fed201e39bac183
Author: Dmitry Tantsur <email address hidden>
Date: Mon Oct 17 14:31:40 2016 +0200

    Only disable Neutron tests when our Grenade is running

    Unconditionally disabling it in code disables neutron for everyone
    importing our plugin.

    Change-Id: I468af14f42b6d9227179e921f5a3ccea2cae0d66
    Partial-Bug: #1621791

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic-inspector (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/389272

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic-inspector (stable/newton)

Reviewed: https://review.openstack.org/389272
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=fbef23ec9b0411cc517a3a1f740932b92c0b7ce0
Submitter: Jenkins
Branch: stable/newton

commit fbef23ec9b0411cc517a3a1f740932b92c0b7ce0
Author: Dmitry Tantsur <email address hidden>
Date: Mon Oct 17 14:31:40 2016 +0200

    Only disable Neutron tests when our Grenade is running

    Unconditionally disabling it in code disables neutron for everyone
    importing our plugin.

    Change-Id: I468af14f42b6d9227179e921f5a3ccea2cae0d66
    Partial-Bug: #1621791
    (cherry picked from commit 4eb77865ab140c7a10e33c0a6fed201e39bac183)

tags: added: in-stable-newton
Changed in ironic-inspector:
assignee: Dmitry Tantsur (divius) → milan k (vetrisko)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic-inspector (master)

Reviewed: https://review.openstack.org/399478
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=25c5a774e31fc229860f35b12e07f67fe2582441
Submitter: Jenkins
Branch: master

commit 25c5a774e31fc229860f35b12e07f67fe2582441
Author: Anton Arefiev <email address hidden>
Date: Fri Nov 18 11:48:07 2016 +0200

    Stop disabling the Neutron tempest plugin

    The root cause for the Inspector grenade failures was an eventlet
    monkey patch[1]. The issue now being solved, disabling the Neutron
    tempest plugin is no longer necessary.

    [1] https://review.openstack.org/#/c/399469/

    Closes-bug: #1621791
    Change-Id: Icaca019880054e1d89b81736c2815a222e8abc45

Changed in ironic-inspector:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic-inspector (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/400714

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic-inspector (stable/newton)

Reviewed: https://review.openstack.org/400714
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=4f920680bba1e91f68a184072763266ab0bcdc81
Submitter: Jenkins
Branch: stable/newton

commit 4f920680bba1e91f68a184072763266ab0bcdc81
Author: Anton Arefiev <email address hidden>
Date: Fri Nov 18 11:48:07 2016 +0200

    Stop disabling the Neutron tempest plugin

    The root cause for the Inspector grenade failures was an eventlet
    monkey patch[1]. The issue now being solved, disabling the Neutron
    tempest plugin is no longer necessary.

    [1] https://review.openstack.org/#/c/399469/

    Closes-bug: #1621791
    Change-Id: Icaca019880054e1d89b81736c2815a222e8abc45
    (cherry picked from commit 25c5a774e31fc229860f35b12e07f67fe2582441)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ironic-inspector 5.0.0

This issue was fixed in the openstack/ironic-inspector 5.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ironic-inspector 4.2.2

This issue was fixed in the openstack/ironic-inspector 4.2.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.