Port list returns HTTP 400 if a node is removed while listing ports

Bug #1748893 reported by Dmitry Tantsur on 2018-02-12
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Fix Released
High
Dmitry Tantsur
Ironic Inspector
Fix Released
Critical
Dmitry Tantsur
Dmitry Tantsur (divius) wrote :

As it can be seen in the logs, it crashes ironic-inspector.

Changed in ironic-inspector:
status: New → Triaged
importance: Undecided → Critical

Fix proposed to branch: master
Review: https://review.openstack.org/543928

Changed in ironic-inspector:
assignee: nobody → Dmitry Tantsur (divius)
status: Triaged → In Progress

Reviewed: https://review.openstack.org/543928
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=3237511cc630c0f1da4cb89a71ab2d945080bc7c
Submitter: Zuul
Branch: master

commit 3237511cc630c0f1da4cb89a71ab2d945080bc7c
Author: Dmitry Tantsur <email address hidden>
Date: Tue Feb 13 14:24:36 2018 +0100

    Retry port lists on failure in PXE filter periodic sync

    These calls are subject to transient network problems, we should
    not abort ironic-inspector process in this case. Also due to
    bug 1748893 the port listing API can sometimes return HTTP 400.

    This change retries port listing 5 times with 1 second break
    before aborting the periodic task and thus the process.

    This change introduces a dependency on the retrying library,
    which is already widely used in OpenStack (including ironic).

    Change-Id: I92fd70ca5692ce9f6798eedf9e540d5aa7c6f1af
    Closes-Bug: #1748893

Changed in ironic-inspector:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/544721
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=a8d621f20a0dbc8db2acafd97c027d066b737b69
Submitter: Zuul
Branch: stable/queens

commit a8d621f20a0dbc8db2acafd97c027d066b737b69
Author: Dmitry Tantsur <email address hidden>
Date: Tue Feb 13 14:24:36 2018 +0100

    Retry port lists on failure in PXE filter periodic sync

    These calls are subject to transient network problems, we should
    not abort ironic-inspector process in this case. Also due to
    bug 1748893 the port listing API can sometimes return HTTP 400.

    This change retries port listing 5 times with 1 second break
    before aborting the periodic task and thus the process.

    This change introduces a dependency on the retrying library,
    which is already widely used in OpenStack (including ironic).

    Change-Id: I92fd70ca5692ce9f6798eedf9e540d5aa7c6f1af
    Closes-Bug: #1748893
    (cherry picked from commit 3237511cc630c0f1da4cb89a71ab2d945080bc7c)

tags: added: in-stable-queens

This issue was fixed in the openstack/ironic-inspector 7.2.0 release.

Fix proposed to branch: master
Review: https://review.openstack.org/545069

Changed in ironic:
status: Triaged → In Progress

Reviewed: https://review.openstack.org/545069
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=52dcc642d372e23fd59be44e0f9f5627fac5cec4
Submitter: Zuul
Branch: master

commit 52dcc642d372e23fd59be44e0f9f5627fac5cec4
Author: Dmitry Tantsur <email address hidden>
Date: Thu Feb 15 18:49:12 2018 +0100

    Fix rare HTTP 400 from port list API

    This may happen when a node is deleted in parallel with calling
    the port list API. Ports are fetched, then we try do fetch their
    nodes and port groups. If either of them are removed in the meantime,
    the API fails with HTTP 400. This change works around it.

    Change-Id: Ie2d4c46c031ee86976abb6107433cdde87a4345a
    Closes-Bug: #1748893

Changed in ironic:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/545892
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=ae956dcba8cafd7c3d0c07836f83fdc70e5c5c86
Submitter: Zuul
Branch: stable/queens

commit ae956dcba8cafd7c3d0c07836f83fdc70e5c5c86
Author: Dmitry Tantsur <email address hidden>
Date: Thu Feb 15 18:49:12 2018 +0100

    Fix rare HTTP 400 from port list API

    This may happen when a node is deleted in parallel with calling
    the port list API. Ports are fetched, then we try do fetch their
    nodes and port groups. If either of them are removed in the meantime,
    the API fails with HTTP 400. This change works around it.

    Change-Id: Ie2d4c46c031ee86976abb6107433cdde87a4345a
    Closes-Bug: #1748893
    (cherry picked from commit 52dcc642d372e23fd59be44e0f9f5627fac5cec4)

This issue was fixed in the openstack/ironic 10.1.1 release.

This issue was fixed in the openstack/ironic-inspector 7.3.0 release.

This issue was fixed in the openstack/ironic 11.0.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers