[Fullstack] TestLegacyL3Agent.test_mtu_update fails sometimes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
Fix Released
|
Critical
|
Slawek Kaplonski |
Bug Description
Traceback (most recent call last):
File "/home/
return f(self, *args, **kwargs)
File "/home/
common_
File "/home/
raise WaitTimeout(
neutron.
So router interface device MTU is not updated after network MTU update.
Changed in neutron: | |
importance: | Undecided → Critical |
Changed in neutron: | |
assignee: | nobody → Slawek Kaplonski (slaweq) |
status: | New → Confirmed |
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master) | #1 |
Slawek Kaplonski (slaweq) wrote : | #2 |
I checked it today and I think I know what is going on there. It's race between processing new ports when interface is added to the router and updating ports when network's mtu was changed.
In failed case, it seems that port is added to the internal cache https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master) | #4 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: master
commit 5c9a7fe1b400c35
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200
Add extra logs to the network update callback in L3 agent
It may be useful during debugging some L3 and events related issues.
Related-bug: #1933234
Change-Id: I4bcba0ae82d99f
Slawek Kaplonski (slaweq) wrote : | #5 |
I just confirmed it with logs from https:/
The issue is that sometimes update_network event comes when router is still processed and port isn't added to the internal cache in https:/
Changed in neutron: | |
status: | Confirmed → Triaged |
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master) | #6 |
Fix proposed to branch: master
Review: https:/
Changed in neutron: | |
status: | Triaged → In Progress |
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master) | #7 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: master
commit 6ce48c30bdc153e
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200
[L3] Use processing queue for network update events
Router_info's _process_
manipulating router_
method from the L3 agent is relying on that Router_info's cache to
check if updated network is connected to the router or not.
So they shouldn't be run together as that may cause some race conditions
and unexpected issues, like e.g. described in the related bug.
Until now, network_update event was the only one which was processed
without using queue of events. And because of that such race condition
as described above were possible.
To fix that, this patch changes network_update method in the way that it
now adds update events for each router hosted by agent to the queue.
Those events for single routers are then processed, checks if network is
actually connected to the router and if yes, schedules router update to
be processed.
Closes-Bug: #1933234
Change-Id: I2efe66a7415f7a
Changed in neutron: | |
status: | In Progress → Fix Released |
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/wallaby) | #8 |
Related fix proposed to branch: stable/wallaby
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby) | #9 |
Fix proposed to branch: stable/wallaby
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/victoria) | #10 |
Related fix proposed to branch: stable/victoria
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria) | #11 |
Fix proposed to branch: stable/victoria
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/ussuri) | #12 |
Related fix proposed to branch: stable/ussuri
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri) | #13 |
Fix proposed to branch: stable/ussuri
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/train) | #14 |
Related fix proposed to branch: stable/train
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train) | #15 |
Fix proposed to branch: stable/train
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/stein) | #16 |
Related fix proposed to branch: stable/stein
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein) | #17 |
Fix proposed to branch: stable/stein
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/rocky) | #18 |
Related fix proposed to branch: stable/rocky
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky) | #19 |
Fix proposed to branch: stable/rocky
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/queens) | #20 |
Related fix proposed to branch: stable/queens
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens) | #21 |
Fix proposed to branch: stable/queens
Review: https:/
tags: | added: neutron-proactive-backport-potential |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/victoria) | #22 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/victoria
commit 7de9aa109a343c8
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200
Add extra logs to the network update callback in L3 agent
It may be useful during debugging some L3 and events related issues.
Related-bug: #1933234
Change-Id: I4bcba0ae82d99f
(cherry picked from commit 5c9a7fe1b400c35
(cherry picked from commit 25f4864d1206bd0
tags: | added: in-stable-victoria |
tags: | added: in-stable-ussuri |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ussuri) | #23 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/ussuri
commit 5dfd2e561556fc1
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200
Add extra logs to the network update callback in L3 agent
It may be useful during debugging some L3 and events related issues.
Related-bug: #1933234
Change-Id: I4bcba0ae82d99f
(cherry picked from commit 5c9a7fe1b400c35
(cherry picked from commit 25f4864d1206bd0
(cherry picked from commit 7de9aa109a343c8
tags: | added: in-stable-train |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/train) | #24 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/train
commit 85c33d4ec9fa81d
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200
Add extra logs to the network update callback in L3 agent
It may be useful during debugging some L3 and events related issues.
Related-bug: #1933234
Change-Id: I4bcba0ae82d99f
(cherry picked from commit 5c9a7fe1b400c35
(cherry picked from commit 25f4864d1206bd0
(cherry picked from commit 7de9aa109a343c8
(cherry picked from commit 5dfd2e561556fc1
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/wallaby) | #25 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/wallaby
commit 25f4864d1206bd0
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200
Add extra logs to the network update callback in L3 agent
It may be useful during debugging some L3 and events related issues.
Related-bug: #1933234
Change-Id: I4bcba0ae82d99f
(cherry picked from commit 5c9a7fe1b400c35
tags: | added: in-stable-wallaby |
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.0.0.0rc1 | #26 |
This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/queens) | #27 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/queens
commit 951ba752256802e
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200
Add extra logs to the network update callback in L3 agent
It may be useful during debugging some L3 and events related issues.
Related-bug: #1933234
Change-Id: I4bcba0ae82d99f
(cherry picked from commit 5c9a7fe1b400c35
(cherry picked from commit 25f4864d1206bd0
(cherry picked from commit 7de9aa109a343c8
(cherry picked from commit 5dfd2e561556fc1
(cherry picked from commit 85c33d4ec9fa81d
(cherry picked from commit 7aab77d2084533b
(cherry picked from commit 49ee2e098fad9d8
tags: | added: in-stable-queens |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/stein) | #28 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/stein
commit 7aab77d2084533b
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200
Add extra logs to the network update callback in L3 agent
It may be useful during debugging some L3 and events related issues.
Related-bug: #1933234
Change-Id: I4bcba0ae82d99f
(cherry picked from commit 5c9a7fe1b400c35
(cherry picked from commit 25f4864d1206bd0
(cherry picked from commit 7de9aa109a343c8
(cherry picked from commit 5dfd2e561556fc1
(cherry picked from commit 85c33d4ec9fa81d
tags: | added: in-stable-stein |
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train) | #29 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/train
commit 6bd15da7ea77ec1
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200
[L3] Use processing queue for network update events
Router_info's _process_
manipulating router_
method from the L3 agent is relying on that Router_info's cache to
check if updated network is connected to the router or not.
So they shouldn't be run together as that may cause some race conditions
and unexpected issues, like e.g. described in the related bug.
Until now, network_update event was the only one which was processed
without using queue of events. And because of that such race condition
as described above were possible.
To fix that, this patch changes network_update method in the way that it
now adds update events for each router hosted by agent to the queue.
Those events for single routers are then processed, checks if network is
actually connected to the router and if yes, schedules router update to
be processed.
Conflicts:
Closes-Bug: #1933234
Change-Id: I2efe66a7415f7a
(cherry picked from commit 6ce48c30bdc153e
(cherry picked from commit 604b055c71c374b
(cherry picked from commit 05d0bc6d500aaaf
(cherry picked from commit 463083c71387b24
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein) | #30 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/stein
commit c88793e6d8c59b6
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200
[L3] Use processing queue for network update events
Router_info's _process_
manipulating router_
method from the L3 agent is relying on that Router_info's cache to
check if updated network is connected to the router or not.
So they shouldn't be run together as that may cause some race conditions
and unexpected issues, like e.g. described in the related bug.
Until now, network_update event was the only one which was processed
without using queue of events. And because of that such race condition
as described above were possible.
To fix that, this patch changes network_update method in the way that it
now adds update events for each router hosted by agent to the queue.
Those events for single routers are then processed, checks if network is
actually connected to the router and if yes, schedules router update to
be processed.
Conflicts:
Closes-Bug: #1933234
Change-Id: I2efe66a7415f7a
(cherry picked from commit 6ce48c30bdc153e
(cherry picked from commit 604b055c71c374b
(cherry picked from commit 05d0bc6d500aaaf
(cherry picked from commit 463083c71387b24
(cherry picked from commit 6bd15da7ea77ec1
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby) | #31 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/wallaby
commit 604b055c71c374b
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200
[L3] Use processing queue for network update events
Router_info's _process_
manipulating router_
method from the L3 agent is relying on that Router_info's cache to
check if updated network is connected to the router or not.
So they shouldn't be run together as that may cause some race conditions
and unexpected issues, like e.g. described in the related bug.
Until now, network_update event was the only one which was processed
without using queue of events. And because of that such race condition
as described above were possible.
To fix that, this patch changes network_update method in the way that it
now adds update events for each router hosted by agent to the queue.
Those events for single routers are then processed, checks if network is
actually connected to the router and if yes, schedules router update to
be processed.
Conflicts:
Closes-Bug: #1933234
Change-Id: I2efe66a7415f7a
(cherry picked from commit 6ce48c30bdc153e
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri) | #32 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/ussuri
commit 463083c71387b24
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200
[L3] Use processing queue for network update events
Router_info's _process_
manipulating router_
method from the L3 agent is relying on that Router_info's cache to
check if updated network is connected to the router or not.
So they shouldn't be run together as that may cause some race conditions
and unexpected issues, like e.g. described in the related bug.
Until now, network_update event was the only one which was processed
without using queue of events. And because of that such race condition
as described above were possible.
To fix that, this patch changes network_update method in the way that it
now adds update events for each router hosted by agent to the queue.
Those events for single routers are then processed, checks if network is
actually connected to the router and if yes, schedules router update to
be processed.
Conflicts:
Closes-Bug: #1933234
Change-Id: I2efe66a7415f7a
(cherry picked from commit 6ce48c30bdc153e
(cherry picked from commit 604b055c71c374b
(cherry picked from commit 05d0bc6d500aaaf
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/rocky) | #33 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/rocky
commit 49ee2e098fad9d8
Author: Slawek Kaplonski <email address hidden>
Date: Tue Jun 29 13:39:22 2021 +0200
Add extra logs to the network update callback in L3 agent
It may be useful during debugging some L3 and events related issues.
Related-bug: #1933234
Change-Id: I4bcba0ae82d99f
(cherry picked from commit 5c9a7fe1b400c35
(cherry picked from commit 25f4864d1206bd0
(cherry picked from commit 7de9aa109a343c8
(cherry picked from commit 5dfd2e561556fc1
(cherry picked from commit 85c33d4ec9fa81d
(cherry picked from commit 7aab77d2084533b
tags: | added: in-stable-rocky |
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky) | #34 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/rocky
commit e2d5d65a4120942
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200
[L3] Use processing queue for network update events
Router_info's _process_
manipulating router_
method from the L3 agent is relying on that Router_info's cache to
check if updated network is connected to the router or not.
So they shouldn't be run together as that may cause some race conditions
and unexpected issues, like e.g. described in the related bug.
Until now, network_update event was the only one which was processed
without using queue of events. And because of that such race condition
as described above were possible.
To fix that, this patch changes network_update method in the way that it
now adds update events for each router hosted by agent to the queue.
Those events for single routers are then processed, checks if network is
actually connected to the router and if yes, schedules router update to
be processed.
Conflicts:
Closes-Bug: #1933234
Change-Id: I2efe66a7415f7a
(cherry picked from commit 6ce48c30bdc153e
(cherry picked from commit 604b055c71c374b
(cherry picked from commit 05d0bc6d500aaaf
(cherry picked from commit 463083c71387b24
(cherry picked from commit 6bd15da7ea77ec1
(cherry picked from commit c88793e6d8c59b6
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria) | #35 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/victoria
commit 05d0bc6d500aaaf
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200
[L3] Use processing queue for network update events
Router_info's _process_
manipulating router_
method from the L3 agent is relying on that Router_info's cache to
check if updated network is connected to the router or not.
So they shouldn't be run together as that may cause some race conditions
and unexpected issues, like e.g. described in the related bug.
Until now, network_update event was the only one which was processed
without using queue of events. And because of that such race condition
as described above were possible.
To fix that, this patch changes network_update method in the way that it
now adds update events for each router hosted by agent to the queue.
Those events for single routers are then processed, checks if network is
actually connected to the router and if yes, schedules router update to
be processed.
Conflicts:
Closes-Bug: #1933234
Change-Id: I2efe66a7415f7a
(cherry picked from commit 6ce48c30bdc153e
(cherry picked from commit 604b055c71c374b
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens) | #36 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/queens
commit e950a57c1123dde
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 7 12:00:14 2021 +0200
[L3] Use processing queue for network update events
Router_info's _process_
manipulating router_
method from the L3 agent is relying on that Router_info's cache to
check if updated network is connected to the router or not.
So they shouldn't be run together as that may cause some race conditions
and unexpected issues, like e.g. described in the related bug.
Until now, network_update event was the only one which was processed
without using queue of events. And because of that such race condition
as described above were possible.
To fix that, this patch changes network_update method in the way that it
now adds update events for each router hosted by agent to the queue.
Those events for single routers are then processed, checks if network is
actually connected to the router and if yes, schedules router update to
be processed.
Conflicts:
Closes-Bug: #1933234
Change-Id: I2efe66a7415f7a
(cherry picked from commit 6ce48c30bdc153e
(cherry picked from commit 604b055c71c374b
(cherry picked from commit 05d0bc6d500aaaf
(cherry picked from commit 463083c71387b24
(cherry picked from commit 6bd15da7ea77ec1
(cherry picked from commit c88793e6d8c59b6
(cherry picked from commit e2d5d65a4120942
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.4.2 | #37 |
This issue was fixed in the openstack/neutron 16.4.2 release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.3.0 | #38 |
This issue was fixed in the openstack/neutron 17.3.0 release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.2.0 | #39 |
This issue was fixed in the openstack/neutron 18.2.0 release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron queens-eol | #40 |
This issue was fixed in the openstack/neutron queens-eol release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron rocky-eol | #41 |
This issue was fixed in the openstack/neutron rocky-eol release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron stein-eol | #42 |
This issue was fixed in the openstack/neutron stein-eol release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron train-eol | #43 |
This issue was fixed in the openstack/neutron train-eol release.
Related fix proposed to branch: master /review. opendev. org/c/openstack /neutron/ +/798648
Review: https:/