neutron

[QoS] Race condition during port qos rules deletion

Bug #1884512 reported by Rodolfo Alonso
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Undecided
Rodolfo Alonso

Bug Description

Deployment using OVS as network backend. QoS enabled.

During the port deletion, the QoS OVS agent extension removes the possible QoS rules applied on a port (max BW egress/ingress, min BW egress/ingress).

During this process, in "delete_ingress_bw_limit_for_port", the port is checked and then, if exists, it is modified to delete the QoS reference (clear "qos" column in "port" DB register).

As seen in [1], the port can be deleted between both operations (retrieval, modification), resulting in an error.

[1]http://paste.openstack.org/show/795047/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/737247

Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
status: New → In Progress
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

BTW, this patch should be backported up to Queens.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/737535

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/737537

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/737247
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e2d1c2869a2c8883157d29cf47d7a5236e63dbc1
Submitter: Zuul
Branch: master

commit e2d1c2869a2c8883157d29cf47d7a5236e63dbc1
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jun 22 09:47:51 2020 +0000

    [OVS] Make QoS OVS agent deletion operations more resilient

    When a Port is deleted, the QoS extension will reset any rule (QoS
    and Queue registers) applied on this port or will reset the
    related Interface policing parameters.

    If the Port and the related Interface are deleted during the QoS
    extension operation, those commands will fail. This patch makes those
    operations more resiliant by not checking the errors when writing on
    the Port or the Interface register.

    Change-Id: I2cc4cdf5be25fab6adbc64acabb3fffebb693fa6
    Closes-Bug: #1884512

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/738171

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/738172

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/738173

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/737535
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=84ac8cf9ffb66d826d9d7875bb0d1addc9120034
Submitter: Zuul
Branch: stable/ussuri

commit 84ac8cf9ffb66d826d9d7875bb0d1addc9120034
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jun 22 09:47:51 2020 +0000

    [OVS] Make QoS OVS agent deletion operations more resilient

    When a Port is deleted, the QoS extension will reset any rule (QoS
    and Queue registers) applied on this port or will reset the
    related Interface policing parameters.

    If the Port and the related Interface are deleted during the QoS
    extension operation, those commands will fail. This patch makes those
    operations more resiliant by not checking the errors when writing on
    the Port or the Interface register.

    Change-Id: I2cc4cdf5be25fab6adbc64acabb3fffebb693fa6
    Closes-Bug: #1884512
    (cherry picked from commit e2d1c2869a2c8883157d29cf47d7a5236e63dbc1)

tags: added: in-stable-ussuri
tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/737537
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3785868bfb387002f4a89d3fa18c20210e8f17db
Submitter: Zuul
Branch: stable/train

commit 3785868bfb387002f4a89d3fa18c20210e8f17db
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jun 22 09:47:51 2020 +0000

    [OVS] Make QoS OVS agent deletion operations more resilient

    When a Port is deleted, the QoS extension will reset any rule (QoS
    and Queue registers) applied on this port or will reset the
    related Interface policing parameters.

    If the Port and the related Interface are deleted during the QoS
    extension operation, those commands will fail. This patch makes those
    operations more resiliant by not checking the errors when writing on
    the Port or the Interface register.

    Change-Id: I2cc4cdf5be25fab6adbc64acabb3fffebb693fa6
    Closes-Bug: #1884512
    (cherry picked from commit e2d1c2869a2c8883157d29cf47d7a5236e63dbc1)
    (cherry picked from commit 84ac8cf9ffb66d826d9d7875bb0d1addc9120034)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/738171
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7edfb0ef4aba338e0225dffda7c8bf0918ef71cb
Submitter: Zuul
Branch: stable/stein

commit 7edfb0ef4aba338e0225dffda7c8bf0918ef71cb
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jun 22 09:47:51 2020 +0000

    [OVS] Make QoS OVS agent deletion operations more resilient

    When a Port is deleted, the QoS extension will reset any rule (QoS
    and Queue registers) applied on this port or will reset the
    related Interface policing parameters.

    If the Port and the related Interface are deleted during the QoS
    extension operation, those commands will fail. This patch makes those
    operations more resiliant by not checking the errors when writing on
    the Port or the Interface register.

    Change-Id: I2cc4cdf5be25fab6adbc64acabb3fffebb693fa6
    Closes-Bug: #1884512
    (cherry picked from commit e2d1c2869a2c8883157d29cf47d7a5236e63dbc1)
    (cherry picked from commit 84ac8cf9ffb66d826d9d7875bb0d1addc9120034)
    (cherry picked from commit 3785868bfb387002f4a89d3fa18c20210e8f17db)

tags: added: in-stable-stein
Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/738173
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=583c0313edb35afdc0afe94914f2e568cfe7a657
Submitter: Zuul
Branch: stable/queens

commit 583c0313edb35afdc0afe94914f2e568cfe7a657
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jun 22 09:47:51 2020 +0000

    [OVS] Make QoS OVS agent deletion operations more resilient

    When a Port is deleted, the QoS extension will reset any rule (QoS
    and Queue registers) applied on this port or will reset the
    related Interface policing parameters.

    If the Port and the related Interface are deleted during the QoS
    extension operation, those commands will fail. This patch makes those
    operations more resiliant by not checking the errors when writing on
    the Port or the Interface register.

    NOTE: this patch is squashed with [1]. That will fix the problem
    with empty "vsctl" transactions when using this OVS DB implementation.

    [1]https://review.opendev.org/#/c/738574/

    Change-Id: I2cc4cdf5be25fab6adbc64acabb3fffebb693fa6
    Closes-Bug: #1884512
    (cherry picked from commit e2d1c2869a2c8883157d29cf47d7a5236e63dbc1)
    (cherry picked from commit 84ac8cf9ffb66d826d9d7875bb0d1addc9120034)
    (cherry picked from commit 3785868bfb387002f4a89d3fa18c20210e8f17db)
    (cherry picked from commit 7edfb0ef4aba338e0225dffda7c8bf0918ef71cb)
    (cherry picked from commit 94271ad8661a36558172d141a84fdd3891786090)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/738172
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=94271ad8661a36558172d141a84fdd3891786090
Submitter: Zuul
Branch: stable/rocky

commit 94271ad8661a36558172d141a84fdd3891786090
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jun 22 09:47:51 2020 +0000

    [OVS] Make QoS OVS agent deletion operations more resilient

    When a Port is deleted, the QoS extension will reset any rule (QoS
    and Queue registers) applied on this port or will reset the
    related Interface policing parameters.

    If the Port and the related Interface are deleted during the QoS
    extension operation, those commands will fail. This patch makes those
    operations more resiliant by not checking the errors when writing on
    the Port or the Interface register.

    NOTE: this patch is squashed with [1]. That will fix the problem
    with empty "vsctl" transactions when using this OVS DB implementation.

    [1]https://review.opendev.org/#/c/738574/

    Change-Id: I2cc4cdf5be25fab6adbc64acabb3fffebb693fa6
    Closes-Bug: #1884512
    (cherry picked from commit e2d1c2869a2c8883157d29cf47d7a5236e63dbc1)
    (cherry picked from commit 84ac8cf9ffb66d826d9d7875bb0d1addc9120034)
    (cherry picked from commit 3785868bfb387002f4a89d3fa18c20210e8f17db)
    (cherry picked from commit 7edfb0ef4aba338e0225dffda7c8bf0918ef71cb)

tags: added: in-stable-rocky
Slawek Kaplonski (slaweq)
tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron queens-eol

This issue was fixed in the openstack/neutron queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron rocky-eol

This issue was fixed in the openstack/neutron rocky-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.