[Fullstack] TestLegacyL3Agent.test_mtu_update fails sometimes

[Oleg Bondarev]

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/test_l3_agent.py", line 322, in test_mtu_update
    common_utils.wait_until_true(lambda: ri_dev.link.mtu == mtu)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 707, in wait_until_true
    raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds

example: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c93/674044/13/check/neutron-fullstack-with-uwsgi/c9334b7/testr_results.html

So router interface device MTU is not updated after network MTU update.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/798648

[Slawek Kaplonski (slaweq)]

I checked it today and I think I know what is going on there. It's race between processing new ports when interface is added to the router and updating ports when network's mtu was changed.

In failed case, it seems that port is added to the internal cache https://github.com/openstack/neutron/blob/master/neutron/agent/l3/router_info.py#L636 after update is processed thus it isn't really added to the list of the internal port and when network_update (https://github.com/openstack/neutron/blob/6196c0873b4af9abd9055bf92478ff05ae090104/neutron/agent/l3/agent.py#L596) is called, the port is not scheduled to be updated.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/798648
Committed: https://opendev.org/openstack/neutron/commit/5c9a7fe1b400c3549ee1cdd8ffd53895d6db1e86
Submitter: "Zuul (22348)"
Branch: master

commit 5c9a7fe1b400c3549ee1cdd8ffd53895d6db1e86
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200

    Add extra logs to the network update callback in L3 agent

    It may be useful during debugging some L3 and events related issues.

    Related-bug: #1933234
    Change-Id: I4bcba0ae82d99fac962d758b48b1727f344ec7bb

[Slawek Kaplonski (slaweq)]

I just confirmed it with logs from https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bb7/798648/1/gate/neutron-fullstack-with-uwsgi/bb732a9/controller/logs/dsvm-fullstack-logs/TestLegacyL3Agent.test_mtu_update/neutron-l3-agent--2021-07-01--00-21-48-384074_log.txt
The issue is that sometimes update_network event comes when router is still processed and port isn't added to the internal cache in https://github.com/openstack/neutron/blob/3764969b82c6e7b8c74172a1ec4d230ce4ddedcc/neutron/agent/l3/router_info.py#L636 so it's not available in https://github.com/openstack/neutron/blob/3764969b82c6e7b8c74172a1ec4d230ce4ddedcc/neutron/agent/l3/agent.py#L602 thus router is not processed as a result of network_update.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/799781

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/799781
Committed: https://opendev.org/openstack/neutron/commit/6ce48c30bdc153e75c4b9b409203f472b065c11b
Submitter: "Zuul (22348)"
Branch: master

commit 6ce48c30bdc153e75c4b9b409203f472b065c11b
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200

    [L3] Use processing queue for network update events

    Router_info's _process_internal_ports() method is the one which is
    manipulating router_info.internal_ports cache and network_update()
    method from the L3 agent is relying on that Router_info's cache to
    check if updated network is connected to the router or not.
    So they shouldn't be run together as that may cause some race conditions
    and unexpected issues, like e.g. described in the related bug.

    Until now, network_update event was the only one which was processed
    without using queue of events. And because of that such race condition
    as described above were possible.
    To fix that, this patch changes network_update method in the way that it
    now adds update events for each router hosted by agent to the queue.
    Those events for single routers are then processed, checks if network is
    actually connected to the router and if yes, schedules router update to
    be processed.

    Closes-Bug: #1933234
    Change-Id: I2efe66a7415f7a18fb85bd2536a1901e751d6203

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/803571

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/803572

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/803573

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/803574

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/803575

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/803576

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/803577

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/803578

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/803579

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/803580

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/803581

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/803582

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/803583

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/803584

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803573
Committed: https://opendev.org/openstack/neutron/commit/7de9aa109a343c83fe6ae0dbfd9fb7e38a99cf31
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 7de9aa109a343c83fe6ae0dbfd9fb7e38a99cf31
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200

    Add extra logs to the network update callback in L3 agent

    It may be useful during debugging some L3 and events related issues.

    Related-bug: #1933234
    Change-Id: I4bcba0ae82d99fac962d758b48b1727f344ec7bb
    (cherry picked from commit 5c9a7fe1b400c3549ee1cdd8ffd53895d6db1e86)
    (cherry picked from commit 25f4864d1206bd087ac98b4806111701bc96cf77)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803575
Committed: https://opendev.org/openstack/neutron/commit/5dfd2e561556fc187d28e65039898d472e50b61d
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 5dfd2e561556fc187d28e65039898d472e50b61d
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200

    Add extra logs to the network update callback in L3 agent

    It may be useful during debugging some L3 and events related issues.

    Related-bug: #1933234
    Change-Id: I4bcba0ae82d99fac962d758b48b1727f344ec7bb
    (cherry picked from commit 5c9a7fe1b400c3549ee1cdd8ffd53895d6db1e86)
    (cherry picked from commit 25f4864d1206bd087ac98b4806111701bc96cf77)
    (cherry picked from commit 7de9aa109a343c83fe6ae0dbfd9fb7e38a99cf31)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803577
Committed: https://opendev.org/openstack/neutron/commit/85c33d4ec9fa81df2d058eacbf6bb1a37039358f
Submitter: "Zuul (22348)"
Branch: stable/train

commit 85c33d4ec9fa81df2d058eacbf6bb1a37039358f
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200

    Add extra logs to the network update callback in L3 agent

    It may be useful during debugging some L3 and events related issues.

    Related-bug: #1933234
    Change-Id: I4bcba0ae82d99fac962d758b48b1727f344ec7bb
    (cherry picked from commit 5c9a7fe1b400c3549ee1cdd8ffd53895d6db1e86)
    (cherry picked from commit 25f4864d1206bd087ac98b4806111701bc96cf77)
    (cherry picked from commit 7de9aa109a343c83fe6ae0dbfd9fb7e38a99cf31)
    (cherry picked from commit 5dfd2e561556fc187d28e65039898d472e50b61d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803571
Committed: https://opendev.org/openstack/neutron/commit/25f4864d1206bd087ac98b4806111701bc96cf77
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 25f4864d1206bd087ac98b4806111701bc96cf77
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200

    Add extra logs to the network update callback in L3 agent

    It may be useful during debugging some L3 and events related issues.

    Related-bug: #1933234
    Change-Id: I4bcba0ae82d99fac962d758b48b1727f344ec7bb
    (cherry picked from commit 5c9a7fe1b400c3549ee1cdd8ffd53895d6db1e86)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803583
Committed: https://opendev.org/openstack/neutron/commit/951ba752256802eaed2780aec1b5a42dcf23a417
Submitter: "Zuul (22348)"
Branch: stable/queens

commit 951ba752256802eaed2780aec1b5a42dcf23a417
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200

    Add extra logs to the network update callback in L3 agent

    It may be useful during debugging some L3 and events related issues.

    Related-bug: #1933234
    Change-Id: I4bcba0ae82d99fac962d758b48b1727f344ec7bb
    (cherry picked from commit 5c9a7fe1b400c3549ee1cdd8ffd53895d6db1e86)
    (cherry picked from commit 25f4864d1206bd087ac98b4806111701bc96cf77)
    (cherry picked from commit 7de9aa109a343c83fe6ae0dbfd9fb7e38a99cf31)
    (cherry picked from commit 5dfd2e561556fc187d28e65039898d472e50b61d)
    (cherry picked from commit 85c33d4ec9fa81df2d058eacbf6bb1a37039358f)
    (cherry picked from commit 7aab77d2084533bac7db7bf27eec328192310b29)
    (cherry picked from commit 49ee2e098fad9d8ca1ab1c7e659f3e485d8444d7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803579
Committed: https://opendev.org/openstack/neutron/commit/7aab77d2084533bac7db7bf27eec328192310b29
Submitter: "Zuul (22348)"
Branch: stable/stein

commit 7aab77d2084533bac7db7bf27eec328192310b29
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200

    Add extra logs to the network update callback in L3 agent

    It may be useful during debugging some L3 and events related issues.

    Related-bug: #1933234
    Change-Id: I4bcba0ae82d99fac962d758b48b1727f344ec7bb
    (cherry picked from commit 5c9a7fe1b400c3549ee1cdd8ffd53895d6db1e86)
    (cherry picked from commit 25f4864d1206bd087ac98b4806111701bc96cf77)
    (cherry picked from commit 7de9aa109a343c83fe6ae0dbfd9fb7e38a99cf31)
    (cherry picked from commit 5dfd2e561556fc187d28e65039898d472e50b61d)
    (cherry picked from commit 85c33d4ec9fa81df2d058eacbf6bb1a37039358f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803578
Committed: https://opendev.org/openstack/neutron/commit/6bd15da7ea77ec1d36df99a45eb26a57578f3d15
Submitter: "Zuul (22348)"
Branch: stable/train

commit 6bd15da7ea77ec1d36df99a45eb26a57578f3d15
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200

    [L3] Use processing queue for network update events

    Router_info's _process_internal_ports() method is the one which is
    manipulating router_info.internal_ports cache and network_update()
    method from the L3 agent is relying on that Router_info's cache to
    check if updated network is connected to the router or not.
    So they shouldn't be run together as that may cause some race conditions
    and unexpected issues, like e.g. described in the related bug.

    Until now, network_update event was the only one which was processed
    without using queue of events. And because of that such race condition
    as described above were possible.
    To fix that, this patch changes network_update method in the way that it
    now adds update events for each router hosted by agent to the queue.
    Those events for single routers are then processed, checks if network is
    actually connected to the router and if yes, schedules router update to
    be processed.

    Conflicts:
        neutron/agent/l3/agent.py

    Closes-Bug: #1933234
    Change-Id: I2efe66a7415f7a18fb85bd2536a1901e751d6203
    (cherry picked from commit 6ce48c30bdc153e75c4b9b409203f472b065c11b)
    (cherry picked from commit 604b055c71c374bcf517359c118ace267e3a4ab1)
    (cherry picked from commit 05d0bc6d500aaafe26fd667a761dae408a8c8ef8)
    (cherry picked from commit 463083c71387b249b9aa834e126a17a1a0f3a189)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803580
Committed: https://opendev.org/openstack/neutron/commit/c88793e6d8c59b67bad646d12af2a4fdad85baa0
Submitter: "Zuul (22348)"
Branch: stable/stein

commit c88793e6d8c59b67bad646d12af2a4fdad85baa0
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200

    [L3] Use processing queue for network update events

    Router_info's _process_internal_ports() method is the one which is
    manipulating router_info.internal_ports cache and network_update()
    method from the L3 agent is relying on that Router_info's cache to
    check if updated network is connected to the router or not.
    So they shouldn't be run together as that may cause some race conditions
    and unexpected issues, like e.g. described in the related bug.

    Until now, network_update event was the only one which was processed
    without using queue of events. And because of that such race condition
    as described above were possible.
    To fix that, this patch changes network_update method in the way that it
    now adds update events for each router hosted by agent to the queue.
    Those events for single routers are then processed, checks if network is
    actually connected to the router and if yes, schedules router update to
    be processed.

    Conflicts:
        neutron/agent/l3/agent.py

    Closes-Bug: #1933234
    Change-Id: I2efe66a7415f7a18fb85bd2536a1901e751d6203
    (cherry picked from commit 6ce48c30bdc153e75c4b9b409203f472b065c11b)
    (cherry picked from commit 604b055c71c374bcf517359c118ace267e3a4ab1)
    (cherry picked from commit 05d0bc6d500aaafe26fd667a761dae408a8c8ef8)
    (cherry picked from commit 463083c71387b249b9aa834e126a17a1a0f3a189)
    (cherry picked from commit 6bd15da7ea77ec1d36df99a45eb26a57578f3d15)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803572
Committed: https://opendev.org/openstack/neutron/commit/604b055c71c374bcf517359c118ace267e3a4ab1
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 604b055c71c374bcf517359c118ace267e3a4ab1
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200

    [L3] Use processing queue for network update events

    Router_info's _process_internal_ports() method is the one which is
    manipulating router_info.internal_ports cache and network_update()
    method from the L3 agent is relying on that Router_info's cache to
    check if updated network is connected to the router or not.
    So they shouldn't be run together as that may cause some race conditions
    and unexpected issues, like e.g. described in the related bug.

    Until now, network_update event was the only one which was processed
    without using queue of events. And because of that such race condition
    as described above were possible.
    To fix that, this patch changes network_update method in the way that it
    now adds update events for each router hosted by agent to the queue.
    Those events for single routers are then processed, checks if network is
    actually connected to the router and if yes, schedules router update to
    be processed.

    Conflicts:
        neutron/agent/l3/agent.py

    Closes-Bug: #1933234
    Change-Id: I2efe66a7415f7a18fb85bd2536a1901e751d6203
    (cherry picked from commit 6ce48c30bdc153e75c4b9b409203f472b065c11b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803576
Committed: https://opendev.org/openstack/neutron/commit/463083c71387b249b9aa834e126a17a1a0f3a189
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 463083c71387b249b9aa834e126a17a1a0f3a189
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200

    [L3] Use processing queue for network update events

    Router_info's _process_internal_ports() method is the one which is
    manipulating router_info.internal_ports cache and network_update()
    method from the L3 agent is relying on that Router_info's cache to
    check if updated network is connected to the router or not.
    So they shouldn't be run together as that may cause some race conditions
    and unexpected issues, like e.g. described in the related bug.

    Until now, network_update event was the only one which was processed
    without using queue of events. And because of that such race condition
    as described above were possible.
    To fix that, this patch changes network_update method in the way that it
    now adds update events for each router hosted by agent to the queue.
    Those events for single routers are then processed, checks if network is
    actually connected to the router and if yes, schedules router update to
    be processed.

    Conflicts:
        neutron/agent/l3/agent.py

    Closes-Bug: #1933234
    Change-Id: I2efe66a7415f7a18fb85bd2536a1901e751d6203
    (cherry picked from commit 6ce48c30bdc153e75c4b9b409203f472b065c11b)
    (cherry picked from commit 604b055c71c374bcf517359c118ace267e3a4ab1)
    (cherry picked from commit 05d0bc6d500aaafe26fd667a761dae408a8c8ef8)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803581
Committed: https://opendev.org/openstack/neutron/commit/49ee2e098fad9d8ca1ab1c7e659f3e485d8444d7
Submitter: "Zuul (22348)"
Branch: stable/rocky

commit 49ee2e098fad9d8ca1ab1c7e659f3e485d8444d7
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200

    Add extra logs to the network update callback in L3 agent

    It may be useful during debugging some L3 and events related issues.

    Related-bug: #1933234
    Change-Id: I4bcba0ae82d99fac962d758b48b1727f344ec7bb
    (cherry picked from commit 5c9a7fe1b400c3549ee1cdd8ffd53895d6db1e86)
    (cherry picked from commit 25f4864d1206bd087ac98b4806111701bc96cf77)
    (cherry picked from commit 7de9aa109a343c83fe6ae0dbfd9fb7e38a99cf31)
    (cherry picked from commit 5dfd2e561556fc187d28e65039898d472e50b61d)
    (cherry picked from commit 85c33d4ec9fa81df2d058eacbf6bb1a37039358f)
    (cherry picked from commit 7aab77d2084533bac7db7bf27eec328192310b29)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803582
Committed: https://opendev.org/openstack/neutron/commit/e2d5d65a41209423e2d980db407b0567f97b903d
Submitter: "Zuul (22348)"
Branch: stable/rocky

commit e2d5d65a41209423e2d980db407b0567f97b903d
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200

    [L3] Use processing queue for network update events

    Router_info's _process_internal_ports() method is the one which is
    manipulating router_info.internal_ports cache and network_update()
    method from the L3 agent is relying on that Router_info's cache to
    check if updated network is connected to the router or not.
    So they shouldn't be run together as that may cause some race conditions
    and unexpected issues, like e.g. described in the related bug.

    Until now, network_update event was the only one which was processed
    without using queue of events. And because of that such race condition
    as described above were possible.
    To fix that, this patch changes network_update method in the way that it
    now adds update events for each router hosted by agent to the queue.
    Those events for single routers are then processed, checks if network is
    actually connected to the router and if yes, schedules router update to
    be processed.

    Conflicts:
        neutron/agent/l3/agent.py
        neutron/tests/unit/agent/l3/test_agent.py

    Closes-Bug: #1933234
    Change-Id: I2efe66a7415f7a18fb85bd2536a1901e751d6203
    (cherry picked from commit 6ce48c30bdc153e75c4b9b409203f472b065c11b)
    (cherry picked from commit 604b055c71c374bcf517359c118ace267e3a4ab1)
    (cherry picked from commit 05d0bc6d500aaafe26fd667a761dae408a8c8ef8)
    (cherry picked from commit 463083c71387b249b9aa834e126a17a1a0f3a189)
    (cherry picked from commit 6bd15da7ea77ec1d36df99a45eb26a57578f3d15)
    (cherry picked from commit c88793e6d8c59b67bad646d12af2a4fdad85baa0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803574
Committed: https://opendev.org/openstack/neutron/commit/05d0bc6d500aaafe26fd667a761dae408a8c8ef8
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 05d0bc6d500aaafe26fd667a761dae408a8c8ef8
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200

    [L3] Use processing queue for network update events

    Router_info's _process_internal_ports() method is the one which is
    manipulating router_info.internal_ports cache and network_update()
    method from the L3 agent is relying on that Router_info's cache to
    check if updated network is connected to the router or not.
    So they shouldn't be run together as that may cause some race conditions
    and unexpected issues, like e.g. described in the related bug.

    Until now, network_update event was the only one which was processed
    without using queue of events. And because of that such race condition
    as described above were possible.
    To fix that, this patch changes network_update method in the way that it
    now adds update events for each router hosted by agent to the queue.
    Those events for single routers are then processed, checks if network is
    actually connected to the router and if yes, schedules router update to
    be processed.

    Conflicts:
        neutron/agent/l3/agent.py

    Closes-Bug: #1933234
    Change-Id: I2efe66a7415f7a18fb85bd2536a1901e751d6203
    (cherry picked from commit 6ce48c30bdc153e75c4b9b409203f472b065c11b)
    (cherry picked from commit 604b055c71c374bcf517359c118ace267e3a4ab1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/803584
Committed: https://opendev.org/openstack/neutron/commit/e950a57c1123ddecc91cc8ecb000d7b9eaaa515e
Submitter: "Zuul (22348)"
Branch: stable/queens

commit e950a57c1123ddecc91cc8ecb000d7b9eaaa515e
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200

    [L3] Use processing queue for network update events

    Router_info's _process_internal_ports() method is the one which is
    manipulating router_info.internal_ports cache and network_update()
    method from the L3 agent is relying on that Router_info's cache to
    check if updated network is connected to the router or not.
    So they shouldn't be run together as that may cause some race conditions
    and unexpected issues, like e.g. described in the related bug.

    Until now, network_update event was the only one which was processed
    without using queue of events. And because of that such race condition
    as described above were possible.
    To fix that, this patch changes network_update method in the way that it
    now adds update events for each router hosted by agent to the queue.
    Those events for single routers are then processed, checks if network is
    actually connected to the router and if yes, schedules router update to
    be processed.

    Conflicts:
        neutron/agent/l3/agent.py
        neutron/tests/unit/agent/l3/test_agent.py
        neutron/agent/common/resource_processing_queue.py

    Closes-Bug: #1933234
    Change-Id: I2efe66a7415f7a18fb85bd2536a1901e751d6203
    (cherry picked from commit 6ce48c30bdc153e75c4b9b409203f472b065c11b)
    (cherry picked from commit 604b055c71c374bcf517359c118ace267e3a4ab1)
    (cherry picked from commit 05d0bc6d500aaafe26fd667a761dae408a8c8ef8)
    (cherry picked from commit 463083c71387b249b9aa834e126a17a1a0f3a189)
    (cherry picked from commit 6bd15da7ea77ec1d36df99a45eb26a57578f3d15)
    (cherry picked from commit c88793e6d8c59b67bad646d12af2a4fdad85baa0)
    (cherry picked from commit e2d5d65a41209423e2d980db407b0567f97b903d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.4.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron stein-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.