neutron

Nova notifications with error response are not retried

Bug #1987780 reported by Szymon Wróblewski
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Szymon Wróblewski

Bug Description

Sometimes Neutron is failing to send notification to Nova due to timeout, refused connection or another HTTP error.
Whenever such event occurs, Neutron notifier for Nova will log an error and consider it's job done.
Meanwhile, since there is no retry mechanism, Nova will wait till timeout and put VM in error state.

Szymon Wróblewski (bluex)
Changed in neutron:
assignee: nobody → Szymon Wróblewski (bluex)
Slawek Kaplonski (slaweq)
Changed in neutron:
importance: Undecided → Medium
Revision history for this message
Herve Beraud (herveberaud) wrote :

Please can you paste the error that you see in the logs?

Do you know if increasing the rpc_response_timeout help to solve this problem?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/855625

Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/855625
Committed: https://opendev.org/openstack/neutron/commit/cd475f9af898b81d98b3e0d3f55b94ea653c193c
Submitter: "Zuul (22348)"
Branch: master

commit cd475f9af898b81d98b3e0d3f55b94ea653c193c
Author: Szymon Wroblewski <email address hidden>
Date: Fri Sep 2 11:26:40 2022 +0200

    Retry connections to Nova

    Sometimes Neutron is failing to send notification to Nova
    due to timeout, refused connection or another HTTP error.
    Retry send in those cases.

    Closes-Bug: #1987780
    Change-Id: Iaaccec770484234b704f70f3c144efac4d8ffba0

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/856578

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/856579

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/856580

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/856606

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/856607

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/856581

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 21.0.0.0rc1

This issue was fixed in the openstack/neutron 21.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/856578
Committed: https://opendev.org/openstack/neutron/commit/e60184b48310e2b74d5b3290ea90fc7aced380ca
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit e60184b48310e2b74d5b3290ea90fc7aced380ca
Author: Szymon Wroblewski <email address hidden>
Date: Fri Sep 2 11:26:40 2022 +0200

    Retry connections to Nova

    Sometimes Neutron is failing to send notification to Nova
    due to timeout, refused connection or another HTTP error.
    Retry send in those cases.

    Closes-Bug: #1987780
    Change-Id: Iaaccec770484234b704f70f3c144efac4d8ffba0
    (cherry picked from commit cd475f9af898b81d98b3e0d3f55b94ea653c193c)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/856579
Committed: https://opendev.org/openstack/neutron/commit/3de47d284e899d1b1d57625a3eabfe4408301e05
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 3de47d284e899d1b1d57625a3eabfe4408301e05
Author: Szymon Wroblewski <email address hidden>
Date: Fri Sep 2 11:26:40 2022 +0200

    Retry connections to Nova

    Sometimes Neutron is failing to send notification to Nova
    due to timeout, refused connection or another HTTP error.
    Retry send in those cases.

    Closes-Bug: #1987780
    Change-Id: Iaaccec770484234b704f70f3c144efac4d8ffba0
    (cherry picked from commit cd475f9af898b81d98b3e0d3f55b94ea653c193c)

tags: added: in-stable-xena
tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/856606
Committed: https://opendev.org/openstack/neutron/commit/49f49bc2bf3b477df00a92e25a8de66221ec8ee6
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 49f49bc2bf3b477df00a92e25a8de66221ec8ee6
Author: Szymon Wroblewski <email address hidden>
Date: Fri Sep 2 11:26:40 2022 +0200

    Retry connections to Nova

    Sometimes Neutron is failing to send notification to Nova
    due to timeout, refused connection or another HTTP error.
    Retry send in those cases.

    Conflicts:
        neutron/notifiers/nova.py
        neutron/tests/unit/notifiers/test_nova.py

    Closes-Bug: #1987780
    Change-Id: Iaaccec770484234b704f70f3c144efac4d8ffba0
    (cherry picked from commit cd475f9af898b81d98b3e0d3f55b94ea653c193c)
    (cherry picked from commit b5e9148cc7311080ba1b2a410145949c1adaa0ca)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/856607
Committed: https://opendev.org/openstack/neutron/commit/3758762021919aa90a0d94b20e8f31049347fe5e
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 3758762021919aa90a0d94b20e8f31049347fe5e
Author: Szymon Wroblewski <email address hidden>
Date: Fri Sep 2 11:26:40 2022 +0200

    Retry connections to Nova

    Sometimes Neutron is failing to send notification to Nova
    due to timeout, refused connection or another HTTP error.
    Retry send in those cases.

    Conflicts:
        neutron/notifiers/nova.py
        neutron/tests/unit/notifiers/test_nova.py

    Closes-Bug: #1987780
    Change-Id: Iaaccec770484234b704f70f3c144efac4d8ffba0
    (cherry picked from commit cd475f9af898b81d98b3e0d3f55b94ea653c193c)
    (cherry picked from commit b5e9148cc7311080ba1b2a410145949c1adaa0ca)
    (cherry picked from commit 49f49bc2bf3b477df00a92e25a8de66221ec8ee6)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/856580
Committed: https://opendev.org/openstack/neutron/commit/b5e9148cc7311080ba1b2a410145949c1adaa0ca
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b5e9148cc7311080ba1b2a410145949c1adaa0ca
Author: Szymon Wroblewski <email address hidden>
Date: Fri Sep 2 11:26:40 2022 +0200

    Retry connections to Nova

    Sometimes Neutron is failing to send notification to Nova
    due to timeout, refused connection or another HTTP error.
    Retry send in those cases.

    Closes-Bug: #1987780
    Change-Id: Iaaccec770484234b704f70f3c144efac4d8ffba0
    (cherry picked from commit cd475f9af898b81d98b3e0d3f55b94ea653c193c)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/856581
Committed: https://opendev.org/openstack/neutron/commit/c9dbbd41921ba5e81c986e78791c5ebfa5816c93
Submitter: "Zuul (22348)"
Branch: stable/train

commit c9dbbd41921ba5e81c986e78791c5ebfa5816c93
Author: Szymon Wroblewski <email address hidden>
Date: Fri Sep 2 11:26:40 2022 +0200

    Retry connections to Nova

    Sometimes Neutron is failing to send notification to Nova
    due to timeout, refused connection or another HTTP error.
    Retry send in those cases.

    Conflicts:
        neutron/notifiers/nova.py
        neutron/tests/unit/notifiers/test_nova.py

    Closes-Bug: #1987780
    Change-Id: Iaaccec770484234b704f70f3c144efac4d8ffba0
    (cherry picked from commit cd475f9af898b81d98b3e0d3f55b94ea653c193c)
    (cherry picked from commit b5e9148cc7311080ba1b2a410145949c1adaa0ca)
    (cherry picked from commit 49f49bc2bf3b477df00a92e25a8de66221ec8ee6)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.6.0

This issue was fixed in the openstack/neutron 18.6.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.5.0

This issue was fixed in the openstack/neutron 19.5.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.3.0

This issue was fixed in the openstack/neutron 20.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron train-eol

This issue was fixed in the openstack/neutron train-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron ussuri-eol

This issue was fixed in the openstack/neutron ussuri-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron victoria-eom

This issue was fixed in the openstack/neutron victoria-eom release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.