CI blocked due to neutron-lib 2.21.0

[Rodolfo Alonso]

The culprit is most likely [1]. I'll push a patch to block this n-lib version in requirements.

Fullstack job is failing: https://zuul.opendev.org/t/openstack/builds?job_name=neutron-fullstack-with-uwsgi&skip=0

Test: neutron.tests.fullstack.test_l3_agent.TestHAL3Agent.test_router_fip_qos_after_admin_state_down_up

Snippet: https://paste.opendev.org/show/bz3yEHLj4MTtF21ap7g1/

Error in neutron-svc: https://8f2d00bef39f83d68021-7883255aaad52a637069aac131629c43.ssl.cf1.rackcdn.com/841396/2/check/neutron-fullstack-with-uwsgi/5331746/controller/logs/dsvm-fullstack-logs/TestHAL3Agent.test_router_fip_qos_after_admin_state_down_up/neutron-server--2022-05-12--09-12-04-498697_log.txt

[1]https://review.opendev.org/c/openstack/neutron-lib/+/828738

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/841576

[Miguel Lavalle (minsel)]

It seems I hit this bug with https://review.opendev.org/c/openstack/neutron/+/837780. Here's the failure. https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c1e/837780/7/check/neutron-fullstack-with-uwsgi/c1e68c6/testr_results.html. Holding my rechecks

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

This bug is related to https://bugs.launchpad.net/neutron/+bug/1942190.

Slawek and I have reviewed the logs of the failing tests. Since [1] and [2], a hidden race condition reproduced in ``test_router_fip_qos_after_admin_state_down_up``. The HA router port is assigned to both L3 agent instances, as part of the virtual router. When the L3 agent calls the "sync_routers" RPC method (``L3PluginApi.get_routers``), the Neutron server calls ``_ensure_host_set_on_ports``. This method will check all router ports and try to bind them to the L3 agent host.

In [3] we can see how the Neutron server receives the update for port "187c9d44-d64f-4a36-9686-1f610c728ae3" from both L3 agents almost at the same time [4]. The Neutron server creates two active port bindings for the same port to the hosts of both L3 agents. This wrong: we can't have two active port bindings for a HA router port.

Slawek will propose a patch.

Regards.

[1]https://review.opendev.org/c/openstack/neutron-lib/+/828738
[2]https://review.opendev.org/c/openstack/neutron/+/833247
[3]https://7e3258951f4b4639a8dc-4c1d5fc7733900a07272f3d252a01439.ssl.cf5.rackcdn.com/841576/1/check/neutron-fullstack-with-uwsgi/1481cf0/controller/logs/dsvm-fullstack-logs/TestHAL3Agent.test_router_fip_qos_after_admin_state_down_up/neutron-server--2022-05-12--12-03-35-353316_log.txt
[4]https://paste.opendev.org/show/bFhVz2gT0QACrwp94HYB/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/841712

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/841712
Committed: https://opendev.org/openstack/neutron/commit/cd8bf18150c8b0a4bc64979d800726483d9cdb6e
Submitter: "Zuul (22348)"
Branch: master

commit cd8bf18150c8b0a4bc64979d800726483d9cdb6e
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 13 11:56:02 2022 +0200

    [L3HA] Don't update HA router's ports if router isn't active on agents

    In case when HA router isn't active on any L3 agent,
    _ensure_host_set_on_port method shouldn't try to update port's host to
    the host from which there was an rpc message sent, as this can be host
    on which router is in the "standby" mode.
    This method should only update port's host to the router's "active_host"
    if there is such active_host found already.

    Depends-On: https://review.opendev.org/c/openstack/requirements/+/841489

    Closes-Bug: #1973162
    Closes-Bug: #1942190
    Change-Id: Ib3945d294601b35f9b268c25841cd284b52c4ca3

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/841576
Reason: Already implemented.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/860893

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/860894

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/860895

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/860896

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/860897

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/860898

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/860899

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/860900

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/860901

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/860893
Committed: https://opendev.org/openstack/neutron/commit/3867f3c872828b34130162b979a2725e1855f6fd
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 3867f3c872828b34130162b979a2725e1855f6fd
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 13 11:56:02 2022 +0200

    [L3HA] Don't update HA router's ports if router isn't active on agents

    In case when HA router isn't active on any L3 agent,
    _ensure_host_set_on_port method shouldn't try to update port's host to
    the host from which there was an rpc message sent, as this can be host
    on which router is in the "standby" mode.
    This method should only update port's host to the router's "active_host"
    if there is such active_host found already.

    Depends-On: https://review.opendev.org/c/openstack/requirements/+/841489

    Closes-Bug: #1973162
    Closes-Bug: #1942190
    Change-Id: Ib3945d294601b35f9b268c25841cd284b52c4ca3
    (cherry picked from commit cd8bf18150c8b0a4bc64979d800726483d9cdb6e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/860894
Committed: https://opendev.org/openstack/neutron/commit/d44fa77319b129c15eb94cc8a3c1b128010a7ef2
Submitter: "Zuul (22348)"
Branch: stable/xena

commit d44fa77319b129c15eb94cc8a3c1b128010a7ef2
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 13 11:56:02 2022 +0200

    [L3HA] Don't update HA router's ports if router isn't active on agents

    In case when HA router isn't active on any L3 agent,
    _ensure_host_set_on_port method shouldn't try to update port's host to
    the host from which there was an rpc message sent, as this can be host
    on which router is in the "standby" mode.
    This method should only update port's host to the router's "active_host"
    if there is such active_host found already.

    Depends-On: https://review.opendev.org/c/openstack/requirements/+/841489

    Closes-Bug: #1973162
    Closes-Bug: #1942190
    Change-Id: Ib3945d294601b35f9b268c25841cd284b52c4ca3
    (cherry picked from commit cd8bf18150c8b0a4bc64979d800726483d9cdb6e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/860896
Committed: https://opendev.org/openstack/neutron/commit/dd2c495a2807200115c4cca3836eaa69b37b6894
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit dd2c495a2807200115c4cca3836eaa69b37b6894
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 13 11:56:02 2022 +0200

    [L3HA] Don't update HA router's ports if router isn't active on agents

    In case when HA router isn't active on any L3 agent,
    _ensure_host_set_on_port method shouldn't try to update port's host to
    the host from which there was an rpc message sent, as this can be host
    on which router is in the "standby" mode.
    This method should only update port's host to the router's "active_host"
    if there is such active_host found already.

    Depends-On: https://review.opendev.org/c/openstack/requirements/+/841489

    Closes-Bug: #1973162
    Closes-Bug: #1942190
    Change-Id: Ib3945d294601b35f9b268c25841cd284b52c4ca3
    (cherry picked from commit cd8bf18150c8b0a4bc64979d800726483d9cdb6e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/860897
Committed: https://opendev.org/openstack/neutron/commit/3e38b1765200b9c385a64f3b1286ccd8eab298fe
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 3e38b1765200b9c385a64f3b1286ccd8eab298fe
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 13 11:56:02 2022 +0200

    [L3HA] Don't update HA router's ports if router isn't active on agents

    In case when HA router isn't active on any L3 agent,
    _ensure_host_set_on_port method shouldn't try to update port's host to
    the host from which there was an rpc message sent, as this can be host
    on which router is in the "standby" mode.
    This method should only update port's host to the router's "active_host"
    if there is such active_host found already.

    Depends-On: https://review.opendev.org/c/openstack/requirements/+/841489

    Closes-Bug: #1973162
    Closes-Bug: #1942190
    Change-Id: Ib3945d294601b35f9b268c25841cd284b52c4ca3
    (cherry picked from commit cd8bf18150c8b0a4bc64979d800726483d9cdb6e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/860898
Committed: https://opendev.org/openstack/neutron/commit/9562726b1b0c4fcd1b3279c1475ce81c64299b36
Submitter: "Zuul (22348)"
Branch: stable/train

commit 9562726b1b0c4fcd1b3279c1475ce81c64299b36
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 13 11:56:02 2022 +0200

    [L3HA] Don't update HA router's ports if router isn't active on agents

    In case when HA router isn't active on any L3 agent,
    _ensure_host_set_on_port method shouldn't try to update port's host to
    the host from which there was an rpc message sent, as this can be host
    on which router is in the "standby" mode.
    This method should only update port's host to the router's "active_host"
    if there is such active_host found already.

    Depends-On: https://review.opendev.org/c/openstack/requirements/+/841489

    Closes-Bug: #1973162
    Closes-Bug: #1942190
    Change-Id: Ib3945d294601b35f9b268c25841cd284b52c4ca3
    (cherry picked from commit cd8bf18150c8b0a4bc64979d800726483d9cdb6e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/860899
Committed: https://opendev.org/openstack/neutron/commit/68c1df17286f48e75eba6ce21151c6787b0c6e3d
Submitter: "Zuul (22348)"
Branch: stable/stein

commit 68c1df17286f48e75eba6ce21151c6787b0c6e3d
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 13 11:56:02 2022 +0200

    [L3HA] Don't update HA router's ports if router isn't active on agents

    In case when HA router isn't active on any L3 agent,
    _ensure_host_set_on_port method shouldn't try to update port's host to
    the host from which there was an rpc message sent, as this can be host
    on which router is in the "standby" mode.
    This method should only update port's host to the router's "active_host"
    if there is such active_host found already.

    Depends-On: https://review.opendev.org/c/openstack/requirements/+/841489

    Closes-Bug: #1973162
    Closes-Bug: #1942190
    Change-Id: Ib3945d294601b35f9b268c25841cd284b52c4ca3
    (cherry picked from commit cd8bf18150c8b0a4bc64979d800726483d9cdb6e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/860895
Committed: https://opendev.org/openstack/neutron/commit/473af223fd7279189f2fdcebb73379272157ba05
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 473af223fd7279189f2fdcebb73379272157ba05
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 13 11:56:02 2022 +0200

    [L3HA] Don't update HA router's ports if router isn't active on agents

    In case when HA router isn't active on any L3 agent,
    _ensure_host_set_on_port method shouldn't try to update port's host to
    the host from which there was an rpc message sent, as this can be host
    on which router is in the "standby" mode.
    This method should only update port's host to the router's "active_host"
    if there is such active_host found already.

    Depends-On: https://review.opendev.org/c/openstack/requirements/+/841489

    Closes-Bug: #1973162
    Closes-Bug: #1942190
    Change-Id: Ib3945d294601b35f9b268c25841cd284b52c4ca3
    (cherry picked from commit cd8bf18150c8b0a4bc64979d800726483d9cdb6e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron stein-eol release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/860900
Reason: This branch of neutron has transitioned to End of Life.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/860901
Reason: This branch of neutron has transitioned to End of Life.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron victoria-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/requirements zed-eom release.