"process_floating_ip_nat_rules_for_centralized_floatingip" should check if self.snat_iptables_manager was initialized

[Rodolfo Alonso]

Environment:
L3 agent configuration: agent_mode=dvr_snat.
The L3 agent is located in a controller node, acting as a DVR edge router (no HA).

Description:
When "process_floating_ip_nat_rules_for_centralized_floatingip" is called, this method should check first if "self.snat_iptables_manager" has been initialized. The method "process_floating_ip_nat_rules_for_centralized_floatingip" is called from:
  <-- DvrEdgeRouter.process_floating_ip_nat_rules
  <-- RouterInfo.process_snat_dnat_for_fip
  <-- RouterInfo.process_external

The method "RouterInfo.process_external" will first call "RouterInfo._process_external_gateway" --> "DvrEdgeRouter.external_gateway_added" --> "DvrEdgeRouter._create_dvr_gateway". This last method initializes the SNAT iptables manager [1] (this code has been around unchanged six years).

However "DvrEdgeRouter.external_gateway_added" is only called if "ex_gw_port" exists. That means if the GW port does not exist, the SNAT iptables manager is None.

Error example (snippet): https://paste.opendev.org/show/809621/

This bug is similar to https://bugs.launchpad.net/neutron/+bug/1560945 (related patch: https://review.opendev.org/c/openstack/neutron/+/296394).

Steps to Reproduce:
(I'm not 100% sure, I still need to check) Create a FIP in a SNAT DVR router without GW port.

Bugzilla reference: https://bugzilla.redhat.com/show_bug.cgi?id=2008155

[1]https://github.com/openstack/neutron/blob/1d450dbddc8c3d34948ab3d9a8346dd491d9cc7c/neutron/agent/l3/dvr_edge_router.py#L196-L198

[Slawek Kaplonski (slaweq)]

I understand the error but I can't understand really how we can get to it. I'm pretty sure it can't be like said in the bug description as You shouldn't be able to create FIP and associate it to some port in the private network if there is no L3 connectivity between that public (FIP) and private (fixed IP) networks.
So either we are not checking something properly in the API level and that is somehow possible or the way to reproduce that is different.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/811318

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hi Slawek:

What is missing in this router is the external GW port. Both networks, the public and the private, have connectivity through the router. But the public network does not have external connectivity because the GW port has not been assigned.

Because of that, the GW port has not been created in the SNAT namespace and the SNAT iptables manager has not been initialized yet.

The customer hit this problem only once, when restarting the controllers. All the three controllers (DVR, non-HA) had the same problem around this time. This problem never happened again. Although I still don't know 100% how this happened, I'll prevent this issue (as done before [1]) preventing any SNAT iptables use if the manager is not initialized.

Regards.

[1]https://review.opendev.org/c/openstack/neutron/+/296394

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/811318
Committed: https://opendev.org/openstack/neutron/commit/f18edfdf450179f6bc8a47f3b143f2701bd93e0e
Submitter: "Zuul (22348)"
Branch: master

commit f18edfdf450179f6bc8a47f3b143f2701bd93e0e
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Sep 27 16:22:45 2021 +0000

    [DVR] Check if SNAT iptables manager is initialized

    Check if SNAT iptables manager is initialized before processing the
    IP NAT rules. If the router never had an external GW port, the DVR
    GW in the SNAT namespace has not been created and the SNAT iptables
    manager has not been initialized.

    In this case, the IP NAT rules for centralized FIPs (to be applied
    on the SNAT namespace) cannot be set.

    Closes-Bug: #1945215
    Change-Id: I426602514805d728f8cd78e42f2b0979b2101089

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/812282

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/812285

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/812282
Committed: https://opendev.org/openstack/neutron/commit/d4ddc9954de2728ebdb069b5d1df71e620144659
Submitter: "Zuul (22348)"
Branch: stable/xena

commit d4ddc9954de2728ebdb069b5d1df71e620144659
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Sep 27 16:22:45 2021 +0000

    [DVR] Check if SNAT iptables manager is initialized

    Check if SNAT iptables manager is initialized before processing the
    IP NAT rules. If the router never had an external GW port, the DVR
    GW in the SNAT namespace has not been created and the SNAT iptables
    manager has not been initialized.

    In this case, the IP NAT rules for centralized FIPs (to be applied
    on the SNAT namespace) cannot be set.

    Closes-Bug: #1945215
    Change-Id: I426602514805d728f8cd78e42f2b0979b2101089
    (cherry picked from commit f18edfdf450179f6bc8a47f3b143f2701bd93e0e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/812285
Committed: https://opendev.org/openstack/neutron/commit/b9143c37e06567b1ce50f9e858c9289984bb8e4b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b9143c37e06567b1ce50f9e858c9289984bb8e4b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Sep 27 16:22:45 2021 +0000

    [DVR] Check if SNAT iptables manager is initialized

    Check if SNAT iptables manager is initialized before processing the
    IP NAT rules. If the router never had an external GW port, the DVR
    GW in the SNAT namespace has not been created and the SNAT iptables
    manager has not been initialized.

    In this case, the IP NAT rules for centralized FIPs (to be applied
    on the SNAT namespace) cannot be set.

    Conflicts:
        neutron/tests/functional/agent/l3/framework.py

    Closes-Bug: #1945215
    Change-Id: I426602514805d728f8cd78e42f2b0979b2101089
    (cherry picked from commit f18edfdf450179f6bc8a47f3b143f2701bd93e0e)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/813205

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/813206

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/813207

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/813206
Committed: https://opendev.org/openstack/neutron/commit/1cbb2d83d17e6cdb91140329b25d1fda72db74a3
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 1cbb2d83d17e6cdb91140329b25d1fda72db74a3
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Sep 27 16:22:45 2021 +0000

    [DVR] Check if SNAT iptables manager is initialized

    Check if SNAT iptables manager is initialized before processing the
    IP NAT rules. If the router never had an external GW port, the DVR
    GW in the SNAT namespace has not been created and the SNAT iptables
    manager has not been initialized.

    In this case, the IP NAT rules for centralized FIPs (to be applied
    on the SNAT namespace) cannot be set.

    Conflicts:
        neutron/tests/functional/agent/l3/framework.py

    Closes-Bug: #1945215
    Change-Id: I426602514805d728f8cd78e42f2b0979b2101089
    (cherry picked from commit f18edfdf450179f6bc8a47f3b143f2701bd93e0e)
    (cherry picked from commit b9143c37e06567b1ce50f9e858c9289984bb8e4b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/813207
Committed: https://opendev.org/openstack/neutron/commit/31f97b5f98ab56d93ea7d6a11d3d79c2d7722a50
Submitter: "Zuul (22348)"
Branch: stable/train

commit 31f97b5f98ab56d93ea7d6a11d3d79c2d7722a50
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Sep 27 16:22:45 2021 +0000

    [DVR] Check if SNAT iptables manager is initialized

    Check if SNAT iptables manager is initialized before processing the
    IP NAT rules. If the router never had an external GW port, the DVR
    GW in the SNAT namespace has not been created and the SNAT iptables
    manager has not been initialized.

    In this case, the IP NAT rules for centralized FIPs (to be applied
    on the SNAT namespace) cannot be set.

    Conflicts:
        neutron/tests/functional/agent/l3/framework.py

    Closes-Bug: #1945215
    Change-Id: I426602514805d728f8cd78e42f2b0979b2101089
    (cherry picked from commit f18edfdf450179f6bc8a47f3b143f2701bd93e0e)
    (cherry picked from commit b9143c37e06567b1ce50f9e858c9289984bb8e4b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/813205
Committed: https://opendev.org/openstack/neutron/commit/41da1a1eb92a86ac1bf318522fde8a29cf9a39ec
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 41da1a1eb92a86ac1bf318522fde8a29cf9a39ec
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Sep 27 16:22:45 2021 +0000

    [DVR] Check if SNAT iptables manager is initialized

    Check if SNAT iptables manager is initialized before processing the
    IP NAT rules. If the router never had an external GW port, the DVR
    GW in the SNAT namespace has not been created and the SNAT iptables
    manager has not been initialized.

    In this case, the IP NAT rules for centralized FIPs (to be applied
    on the SNAT namespace) cannot be set.

    Conflicts:
        neutron/tests/functional/agent/l3/framework.py

    Closes-Bug: #1945215
    Change-Id: I426602514805d728f8cd78e42f2b0979b2101089
    (cherry picked from commit f18edfdf450179f6bc8a47f3b143f2701bd93e0e)
    (cherry picked from commit b9143c37e06567b1ce50f9e858c9289984bb8e4b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.4.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.