[ovn] Memory consumption grows over time due to MAC_Binding entries in SB database
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
Fix Released
|
High
|
Daniel Alvarez |
Bug Description
MAC_Binding entries are used in OVN as a mechanism to learn MAC addresses on logical ports and avoid sending ARP requests to the network.
There is no aging mechanism for these entries [0] and the table can grow indefinitely. In environments with for example large (eg. /16) external networks; OVN may learn a considerable amount of addresses growing the size of the db a lot.
Today, Neutron monitors this table to workaround the lack of aging mechanism and remove the MAC_Binding entries associated to Floating IPs and each neutron-server worker will keep an in-memory copy of such table increasing its memory footprint to several Gigabytes, eventually leading to OOM killers.
[0] https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master) | #1 |
Changed in neutron: | |
status: | New → In Progress |
Changed in neutron: | |
importance: | Undecided → High |
assignee: | nobody → Daniel Alvarez (dalvarezs) |
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master) | #2 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master) | #3 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: master
commit f6c35527698119e
Author: Daniel Alvarez Sanchez <email address hidden>
Date: Thu Oct 7 09:23:21 2021 +0200
[ovn] Stop monitoring the SB MAC_Binding table to reduce mem footprint
The MAC_Binding table in the SB database may grow indefinitely (due
to a lack of an aging mechanism of its entries) and eventually
lead to OOM killers for neutron-server which maintains an in-memory
copy of the database.
In order to stop monitoring this table, this patch is invoking
the ovsdb-client tool to remove the entries associated to
Floating IPs that have just been detached. The execution of this
tool is really fast as it will just invoke a JSON-RPC transact command
which doesn't require downloading the database contents.
In a scale test, the memory consumption of neutron-server dropped
from 75GB to 7GB with this patch.
Closes-Bug: #1946318
Signed-off-by: Daniel Alvarez Sanchez <email address hidden>
Change-Id: Id84bf17953527c
Changed in neutron: | |
status: | In Progress → Fix Released |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master) | #4 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: master
commit a278c5ba789c014
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Oct 12 11:42:39 2021 +0100
[OVN] Tune OVN routers to reduce the mem footprint for ML2/OVN
In order to avoid having a MAC_Binding table explosion and helping
lowering the memory footprint when using ML2/OVN this patch is setting
two options to the OVN routers:
* always_
avoid learning from ARP replies observed in the network. Only the
ARP requests sent by OVN will generate a MAC_Binding entry in the
OVSDB database. For larger broadcasts domains this avoids having a
MAC_Binding table explosion, reduce the DB size and memory footprint
of ML2/OVN.
* dynamic_
pre-
the number of flows, DB size and memory footprint of ML2/OVN.
For more information on these option for core OVN please refer to:
https:/
This patch also includes a new maintenance task to include these options
to existing routers in the system.
Related-Bug: #1946318
Change-Id: I056acdec9b6ee2
Signed-off-by: Lucas Alvares Gomes <email address hidden>
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/ussuri) | #5 |
Related fix proposed to branch: stable/ussuri
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/victoria) | #6 |
Related fix proposed to branch: stable/victoria
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/wallaby) | #7 |
Related fix proposed to branch: stable/wallaby
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/xena) | #8 |
Related fix proposed to branch: stable/xena
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/xena) | #9 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/xena
commit 987f0689ead8cdb
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Oct 12 11:42:39 2021 +0100
[OVN] Tune OVN routers to reduce the mem footprint for ML2/OVN
In order to avoid having a MAC_Binding table explosion and helping
lowering the memory footprint when using ML2/OVN this patch is setting
two options to the OVN routers:
* always_
avoid learning from ARP replies observed in the network. Only the
ARP requests sent by OVN will generate a MAC_Binding entry in the
OVSDB database. For larger broadcasts domains this avoids having a
MAC_Binding table explosion, reduce the DB size and memory footprint
of ML2/OVN.
* dynamic_
pre-
the number of flows, DB size and memory footprint of ML2/OVN.
For more information on these option for core OVN please refer to:
https:/
This patch also includes a new maintenance task to include these options
to existing routers in the system.
Related-Bug: #1946318
Change-Id: I056acdec9b6ee2
Signed-off-by: Lucas Alvares Gomes <email address hidden>
(cherry picked from commit a278c5ba789c014
tags: | added: in-stable-xena |
tags: | added: in-stable-wallaby |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/wallaby) | #10 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/wallaby
commit 9365bbd19ff57b3
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Oct 12 11:42:39 2021 +0100
[OVN] Tune OVN routers to reduce the mem footprint for ML2/OVN
In order to avoid having a MAC_Binding table explosion and helping
lowering the memory footprint when using ML2/OVN this patch is setting
two options to the OVN routers:
* always_
avoid learning from ARP replies observed in the network. Only the
ARP requests sent by OVN will generate a MAC_Binding entry in the
OVSDB database. For larger broadcasts domains this avoids having a
MAC_Binding table explosion, reduce the DB size and memory footprint
of ML2/OVN.
* dynamic_
pre-
the number of flows, DB size and memory footprint of ML2/OVN.
For more information on these option for core OVN please refer to:
https:/
This patch also includes a new maintenance task to include these options
to existing routers in the system.
Related-Bug: #1946318
Change-Id: I056acdec9b6ee2
Signed-off-by: Lucas Alvares Gomes <email address hidden>
(cherry picked from commit a278c5ba789c014
tags: | added: in-stable-victoria |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/victoria) | #11 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/victoria
commit 34c3ec244e12e74
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Oct 12 11:42:39 2021 +0100
[OVN] Tune OVN routers to reduce the mem footprint for ML2/OVN
In order to avoid having a MAC_Binding table explosion and helping
lowering the memory footprint when using ML2/OVN this patch is setting
two options to the OVN routers:
* always_
avoid learning from ARP replies observed in the network. Only the
ARP requests sent by OVN will generate a MAC_Binding entry in the
OVSDB database. For larger broadcasts domains this avoids having a
MAC_Binding table explosion, reduce the DB size and memory footprint
of ML2/OVN.
* dynamic_
pre-
the number of flows, DB size and memory footprint of ML2/OVN.
For more information on these option for core OVN please refer to:
https:/
This patch also includes a new maintenance task to include these options
to existing routers in the system.
Related-Bug: #1946318
Change-Id: I056acdec9b6ee2
Signed-off-by: Lucas Alvares Gomes <email address hidden>
(cherry picked from commit a278c5ba789c014
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ussuri) | #12 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/ussuri
commit e814e93e8ed9268
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Oct 12 11:42:39 2021 +0100
[OVN] Tune OVN routers to reduce the mem footprint for ML2/OVN
In order to avoid having a MAC_Binding table explosion and helping
lowering the memory footprint when using ML2/OVN this patch is setting
two options to the OVN routers:
* always_
avoid learning from ARP replies observed in the network. Only the
ARP requests sent by OVN will generate a MAC_Binding entry in the
OVSDB database. For larger broadcasts domains this avoids having a
MAC_Binding table explosion, reduce the DB size and memory footprint
of ML2/OVN.
* dynamic_
pre-
the number of flows, DB size and memory footprint of ML2/OVN.
For more information on these option for core OVN please refer to:
https:/
This patch also includes a new maintenance task to include these options
to existing routers in the system.
Related-Bug: #1946318
Change-Id: I056acdec9b6ee2
Signed-off-by: Lucas Alvares Gomes <email address hidden>
(cherry picked from commit a278c5ba789c014
tags: | added: in-stable-ussuri |
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri) | #13 |
Fix proposed to branch: stable/ussuri
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria) | #14 |
Fix proposed to branch: stable/victoria
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby) | #15 |
Fix proposed to branch: stable/wallaby
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/xena) | #16 |
Fix proposed to branch: stable/xena
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena) | #17 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/xena
commit 5c9c366376324bc
Author: Daniel Alvarez Sanchez <email address hidden>
Date: Thu Oct 7 09:23:21 2021 +0200
[ovn] Stop monitoring the SB MAC_Binding table to reduce mem footprint
The MAC_Binding table in the SB database may grow indefinitely (due
to a lack of an aging mechanism of its entries) and eventually
lead to OOM killers for neutron-server which maintains an in-memory
copy of the database.
In order to stop monitoring this table, this patch is invoking
the ovsdb-client tool to remove the entries associated to
Floating IPs that have just been detached. The execution of this
tool is really fast as it will just invoke a JSON-RPC transact command
which doesn't require downloading the database contents.
In a scale test, the memory consumption of neutron-server dropped
from 75GB to 7GB with this patch.
Closes-Bug: #1946318
Signed-off-by: Daniel Alvarez Sanchez <email address hidden>
Change-Id: Id84bf17953527c
(cherry picked from commit f6c35527698119e
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby) | #18 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/wallaby
commit 1a9db2168a87d79
Author: Daniel Alvarez Sanchez <email address hidden>
Date: Thu Oct 7 09:23:21 2021 +0200
[ovn] Stop monitoring the SB MAC_Binding table to reduce mem footprint
The MAC_Binding table in the SB database may grow indefinitely (due
to a lack of an aging mechanism of its entries) and eventually
lead to OOM killers for neutron-server which maintains an in-memory
copy of the database.
In order to stop monitoring this table, this patch is invoking
the ovsdb-client tool to remove the entries associated to
Floating IPs that have just been detached. The execution of this
tool is really fast as it will just invoke a JSON-RPC transact command
which doesn't require downloading the database contents.
In a scale test, the memory consumption of neutron-server dropped
from 75GB to 7GB with this patch.
Closes-Bug: #1946318
Conflicts:
neutron/
neutron/
Signed-off-by: Daniel Alvarez Sanchez <email address hidden>
Change-Id: Id84bf17953527c
(cherry picked from commit f6c35527698119e
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria) | #19 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/victoria
commit 3ac446155225ac9
Author: Daniel Alvarez Sanchez <email address hidden>
Date: Thu Oct 7 09:23:21 2021 +0200
[ovn] Stop monitoring the SB MAC_Binding table to reduce mem footprint
The MAC_Binding table in the SB database may grow indefinitely (due
to a lack of an aging mechanism of its entries) and eventually
lead to OOM killers for neutron-server which maintains an in-memory
copy of the database.
In order to stop monitoring this table, this patch is invoking
the ovsdb-client tool to remove the entries associated to
Floating IPs that have just been detached. The execution of this
tool is really fast as it will just invoke a JSON-RPC transact command
which doesn't require downloading the database contents.
In a scale test, the memory consumption of neutron-server dropped
from 75GB to 7GB with this patch.
Closes-Bug: #1946318
Conflicts:
neutron/
neutron/
neutron/
Note: The functional test has been changed for this stable branch
because the update_tables() method from ovsdbapp is not availabie for
this version of Neutron. So I changed the functional test to add the
MAC_Binding entry via ovsdb-client instead of using ovsdbapp to do so.
Signed-off-by: Daniel Alvarez Sanchez <email address hidden>
Change-Id: Id84bf17953527c
(cherry picked from commit f6c35527698119e
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri) | #20 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/ussuri
commit b767825d1329dbb
Author: Daniel Alvarez Sanchez <email address hidden>
Date: Thu Oct 7 09:23:21 2021 +0200
[ovn] Stop monitoring the SB MAC_Binding table to reduce mem footprint
The MAC_Binding table in the SB database may grow indefinitely (due
to a lack of an aging mechanism of its entries) and eventually
lead to OOM killers for neutron-server which maintains an in-memory
copy of the database.
In order to stop monitoring this table, this patch is invoking
the ovsdb-client tool to remove the entries associated to
Floating IPs that have just been detached. The execution of this
tool is really fast as it will just invoke a JSON-RPC transact command
which doesn't require downloading the database contents.
In a scale test, the memory consumption of neutron-server dropped
from 75GB to 7GB with this patch.
Closes-Bug: #1946318
Conflicts:
neutron/
neutron/
neutron/
Note: The functional test has been changed for this stable branch
because the update_tables() method from ovsdbapp is not availabie for
this version of Neutron. So I changed the functional test to add the
MAC_Binding entry via ovsdb-client instead of using ovsdbapp to do so.
Signed-off-by: Daniel Alvarez Sanchez <email address hidden>
Change-Id: Id84bf17953527c
(cherry picked from commit f6c35527698119e
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master) | #21 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.4.2 | #22 |
This issue was fixed in the openstack/neutron 16.4.2 release.
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master) | #23 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: master
commit 7874c576013928c
Author: Daniel Alvarez Sanchez <email address hidden>
Date: Thu Nov 4 14:34:32 2021 +0100
[ovn] Add timeout option to ovsdb-client command
Today, we invoke ovsdb-client to cleanup the MAC_Binding entries
without specifying any timeout. This can lead to workers blocking
forever if there's an issue with the connection to the server.
This patch is adding a timeout parameter to the command line to
prevent this condition.
Closes-Bug: #1948891
Related-Bug: #1946318
Signed-off-by: Daniel Alvarez Sanchez <email address hidden>
Change-Id: Id393cbec31dd64
tags: | added: neutron-proactive-backport-potential |
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/xena) | #24 |
Related fix proposed to branch: stable/xena
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/wallaby) | #25 |
Related fix proposed to branch: stable/wallaby
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/victoria) | #26 |
Related fix proposed to branch: stable/victoria
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/ussuri) | #27 |
Related fix proposed to branch: stable/ussuri
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/xena) | #28 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/xena
commit 6b270bd6f86784e
Author: Daniel Alvarez Sanchez <email address hidden>
Date: Thu Nov 4 14:34:32 2021 +0100
[ovn] Add timeout option to ovsdb-client command
Today, we invoke ovsdb-client to cleanup the MAC_Binding entries
without specifying any timeout. This can lead to workers blocking
forever if there's an issue with the connection to the server.
This patch is adding a timeout parameter to the command line to
prevent this condition.
Closes-Bug: #1948891
Related-Bug: #1946318
Signed-off-by: Daniel Alvarez Sanchez <email address hidden>
Change-Id: Id393cbec31dd64
(cherry picked from commit 7874c576013928c
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/victoria) | #29 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/victoria
commit 37333d3788dc0ab
Author: Daniel Alvarez Sanchez <email address hidden>
Date: Thu Nov 4 14:34:32 2021 +0100
[ovn] Add timeout option to ovsdb-client command
Today, we invoke ovsdb-client to cleanup the MAC_Binding entries
without specifying any timeout. This can lead to workers blocking
forever if there's an issue with the connection to the server.
This patch is adding a timeout parameter to the command line to
prevent this condition.
Closes-Bug: #1948891
Related-Bug: #1946318
Conflicts:
neutron/
Signed-off-by: Daniel Alvarez Sanchez <email address hidden>
Change-Id: Id393cbec31dd64
(cherry picked from commit 7874c576013928c
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/wallaby) | #30 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/wallaby
commit f37e0be349f3492
Author: Daniel Alvarez Sanchez <email address hidden>
Date: Thu Nov 4 14:34:32 2021 +0100
[ovn] Add timeout option to ovsdb-client command
Today, we invoke ovsdb-client to cleanup the MAC_Binding entries
without specifying any timeout. This can lead to workers blocking
forever if there's an issue with the connection to the server.
This patch is adding a timeout parameter to the command line to
prevent this condition.
Closes-Bug: #1948891
Related-Bug: #1946318
Signed-off-by: Daniel Alvarez Sanchez <email address hidden>
Change-Id: Id393cbec31dd64
(cherry picked from commit 7874c576013928c
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ussuri) | #31 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/ussuri
commit 15e2da82c27f64b
Author: Daniel Alvarez Sanchez <email address hidden>
Date: Thu Nov 4 14:34:32 2021 +0100
[ovn] Add timeout option to ovsdb-client command
Today, we invoke ovsdb-client to cleanup the MAC_Binding entries
without specifying any timeout. This can lead to workers blocking
forever if there's an issue with the connection to the server.
This patch is adding a timeout parameter to the command line to
prevent this condition.
Closes-Bug: #1948891
Related-Bug: #1946318
Conflicts:
Signed-off-by: Daniel Alvarez Sanchez <email address hidden>
Change-Id: Id393cbec31dd64
(cherry picked from commit 7874c576013928c
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.1.0 | #32 |
This issue was fixed in the openstack/neutron 19.1.0 release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.3.0 | #33 |
This issue was fixed in the openstack/neutron 17.3.0 release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.2.0 | #34 |
This issue was fixed in the openstack/neutron 18.2.0 release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.0.0.0rc1 | #35 |
This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn train-eol | #36 |
This issue was fixed in the openstack/
Fix proposed to branch: master /review. opendev. org/c/openstack /neutron/ +/812805
Review: https:/