"process_floating_ip_nat_rules_for_centralized_floatingip" should check if self.snat_iptables_manager was initialized

Bug #1945215 reported by Rodolfo Alonso
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Undecided
Rodolfo Alonso

Bug Description

Environment:
L3 agent configuration: agent_mode=dvr_snat.
The L3 agent is located in a controller node, acting as a DVR edge router (no HA).

Description:
When "process_floating_ip_nat_rules_for_centralized_floatingip" is called, this method should check first if "self.snat_iptables_manager" has been initialized. The method "process_floating_ip_nat_rules_for_centralized_floatingip" is called from:
  <-- DvrEdgeRouter.process_floating_ip_nat_rules
  <-- RouterInfo.process_snat_dnat_for_fip
  <-- RouterInfo.process_external

The method "RouterInfo.process_external" will first call "RouterInfo._process_external_gateway" --> "DvrEdgeRouter.external_gateway_added" --> "DvrEdgeRouter._create_dvr_gateway". This last method initializes the SNAT iptables manager [1] (this code has been around unchanged six years).

However "DvrEdgeRouter.external_gateway_added" is only called if "ex_gw_port" exists. That means if the GW port does not exist, the SNAT iptables manager is None.

Error example (snippet): https://paste.opendev.org/show/809621/

This bug is similar to https://bugs.launchpad.net/neutron/+bug/1560945 (related patch: https://review.opendev.org/c/openstack/neutron/+/296394).

Steps to Reproduce:
(I'm not 100% sure, I still need to check) Create a FIP in a SNAT DVR router without GW port.

Bugzilla reference: https://bugzilla.redhat.com/show_bug.cgi?id=2008155

[1]https://github.com/openstack/neutron/blob/1d450dbddc8c3d34948ab3d9a8346dd491d9cc7c/neutron/agent/l3/dvr_edge_router.py#L196-L198

Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
description: updated
description: updated
Revision history for this message
Slawek Kaplonski (slaweq) wrote :

I understand the error but I can't understand really how we can get to it. I'm pretty sure it can't be like said in the bug description as You shouldn't be able to create FIP and associate it to some port in the private network if there is no L3 connectivity between that public (FIP) and private (fixed IP) networks.
So either we are not checking something properly in the API level and that is somehow possible or the way to reproduce that is different.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/811318

Changed in neutron:
status: New → In Progress
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hi Slawek:

What is missing in this router is the external GW port. Both networks, the public and the private, have connectivity through the router. But the public network does not have external connectivity because the GW port has not been assigned.

Because of that, the GW port has not been created in the SNAT namespace and the SNAT iptables manager has not been initialized yet.

The customer hit this problem only once, when restarting the controllers. All the three controllers (DVR, non-HA) had the same problem around this time. This problem never happened again. Although I still don't know 100% how this happened, I'll prevent this issue (as done before [1]) preventing any SNAT iptables use if the manager is not initialized.

Regards.

[1]https://review.opendev.org/c/openstack/neutron/+/296394

description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/811318
Committed: https://opendev.org/openstack/neutron/commit/f18edfdf450179f6bc8a47f3b143f2701bd93e0e
Submitter: "Zuul (22348)"
Branch: master

commit f18edfdf450179f6bc8a47f3b143f2701bd93e0e
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Sep 27 16:22:45 2021 +0000

    [DVR] Check if SNAT iptables manager is initialized

    Check if SNAT iptables manager is initialized before processing the
    IP NAT rules. If the router never had an external GW port, the DVR
    GW in the SNAT namespace has not been created and the SNAT iptables
    manager has not been initialized.

    In this case, the IP NAT rules for centralized FIPs (to be applied
    on the SNAT namespace) cannot be set.

    Closes-Bug: #1945215
    Change-Id: I426602514805d728f8cd78e42f2b0979b2101089

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/812282

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/812285

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/812282
Committed: https://opendev.org/openstack/neutron/commit/d4ddc9954de2728ebdb069b5d1df71e620144659
Submitter: "Zuul (22348)"
Branch: stable/xena

commit d4ddc9954de2728ebdb069b5d1df71e620144659
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Sep 27 16:22:45 2021 +0000

    [DVR] Check if SNAT iptables manager is initialized

    Check if SNAT iptables manager is initialized before processing the
    IP NAT rules. If the router never had an external GW port, the DVR
    GW in the SNAT namespace has not been created and the SNAT iptables
    manager has not been initialized.

    In this case, the IP NAT rules for centralized FIPs (to be applied
    on the SNAT namespace) cannot be set.

    Closes-Bug: #1945215
    Change-Id: I426602514805d728f8cd78e42f2b0979b2101089
    (cherry picked from commit f18edfdf450179f6bc8a47f3b143f2701bd93e0e)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/812285
Committed: https://opendev.org/openstack/neutron/commit/b9143c37e06567b1ce50f9e858c9289984bb8e4b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b9143c37e06567b1ce50f9e858c9289984bb8e4b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Sep 27 16:22:45 2021 +0000

    [DVR] Check if SNAT iptables manager is initialized

    Check if SNAT iptables manager is initialized before processing the
    IP NAT rules. If the router never had an external GW port, the DVR
    GW in the SNAT namespace has not been created and the SNAT iptables
    manager has not been initialized.

    In this case, the IP NAT rules for centralized FIPs (to be applied
    on the SNAT namespace) cannot be set.

    Conflicts:
        neutron/tests/functional/agent/l3/framework.py

    Closes-Bug: #1945215
    Change-Id: I426602514805d728f8cd78e42f2b0979b2101089
    (cherry picked from commit f18edfdf450179f6bc8a47f3b143f2701bd93e0e)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/813205

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/813206

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/813207

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/813206
Committed: https://opendev.org/openstack/neutron/commit/1cbb2d83d17e6cdb91140329b25d1fda72db74a3
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 1cbb2d83d17e6cdb91140329b25d1fda72db74a3
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Sep 27 16:22:45 2021 +0000

    [DVR] Check if SNAT iptables manager is initialized

    Check if SNAT iptables manager is initialized before processing the
    IP NAT rules. If the router never had an external GW port, the DVR
    GW in the SNAT namespace has not been created and the SNAT iptables
    manager has not been initialized.

    In this case, the IP NAT rules for centralized FIPs (to be applied
    on the SNAT namespace) cannot be set.

    Conflicts:
        neutron/tests/functional/agent/l3/framework.py

    Closes-Bug: #1945215
    Change-Id: I426602514805d728f8cd78e42f2b0979b2101089
    (cherry picked from commit f18edfdf450179f6bc8a47f3b143f2701bd93e0e)
    (cherry picked from commit b9143c37e06567b1ce50f9e858c9289984bb8e4b)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/813207
Committed: https://opendev.org/openstack/neutron/commit/31f97b5f98ab56d93ea7d6a11d3d79c2d7722a50
Submitter: "Zuul (22348)"
Branch: stable/train

commit 31f97b5f98ab56d93ea7d6a11d3d79c2d7722a50
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Sep 27 16:22:45 2021 +0000

    [DVR] Check if SNAT iptables manager is initialized

    Check if SNAT iptables manager is initialized before processing the
    IP NAT rules. If the router never had an external GW port, the DVR
    GW in the SNAT namespace has not been created and the SNAT iptables
    manager has not been initialized.

    In this case, the IP NAT rules for centralized FIPs (to be applied
    on the SNAT namespace) cannot be set.

    Conflicts:
        neutron/tests/functional/agent/l3/framework.py

    Closes-Bug: #1945215
    Change-Id: I426602514805d728f8cd78e42f2b0979b2101089
    (cherry picked from commit f18edfdf450179f6bc8a47f3b143f2701bd93e0e)
    (cherry picked from commit b9143c37e06567b1ce50f9e858c9289984bb8e4b)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/813205
Committed: https://opendev.org/openstack/neutron/commit/41da1a1eb92a86ac1bf318522fde8a29cf9a39ec
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 41da1a1eb92a86ac1bf318522fde8a29cf9a39ec
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Sep 27 16:22:45 2021 +0000

    [DVR] Check if SNAT iptables manager is initialized

    Check if SNAT iptables manager is initialized before processing the
    IP NAT rules. If the router never had an external GW port, the DVR
    GW in the SNAT namespace has not been created and the SNAT iptables
    manager has not been initialized.

    In this case, the IP NAT rules for centralized FIPs (to be applied
    on the SNAT namespace) cannot be set.

    Conflicts:
        neutron/tests/functional/agent/l3/framework.py

    Closes-Bug: #1945215
    Change-Id: I426602514805d728f8cd78e42f2b0979b2101089
    (cherry picked from commit f18edfdf450179f6bc8a47f3b143f2701bd93e0e)
    (cherry picked from commit b9143c37e06567b1ce50f9e858c9289984bb8e4b)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.4.2

This issue was fixed in the openstack/neutron 16.4.2 release.

tags: added: neutron-proactive-backport-potential
tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.1.0

This issue was fixed in the openstack/neutron 19.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.3.0

This issue was fixed in the openstack/neutron 17.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.2.0

This issue was fixed in the openstack/neutron 18.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.0.0.0rc1

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers