Unable to update mtu on DVR snat

[Arnaud Morin]

When updating the mtu on a network, the new mtu is correctly set in qrouter-namespace but not in snat-namespace.

Also, a stack trace is reported in l3-agent logs stating that the qr-abcdef interface is not found.

Step by step:

- create a router in DVR mode (I also have HA, but non HA is also affected)
- add an external gw
- add a network to this router

Check the network mtu on the compute on both namespaces:
$ ip netns exec qrouter-9d5352e8-4f40-49a0-a632-9406042478b8 ip l
...
18: qr-e05ab302-1b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450

$ ip netns exec snat-9d5352e8-4f40-49a0-a632-9406042478b8 ip l
...
16: sg-7ab0954c-73: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450

(mtu is 1450)

- change the network mtu:
$ openstack network set --mtu 1400 MyNetwork

Check the network mtu on the compute on both namespaces:
$ ip netns exec qrouter-9d5352e8-4f40-49a0-a632-9406042478b8 ip l
...
18: qr-e05ab302-1b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400

$ ip netns exec snat-9d5352e8-4f40-49a0-a632-9406042478b8 ip l
...
16: sg-7ab0954c-73: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450

In neutron-l3-agent.log file, I can see:
ERROR neutron.agent.l3.router_info [-] Network interface qr-e05ab302-1b not found in namespace snat-9d5352e8-4f40-49a0-a632-9406042478b8.: neutron.privileged.agent.linux.ip_lib.Net[..truncated..]
ERROR neutron.agent.l3.router_info Traceback (most recent call last):
ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 167, in call
ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 1243, in process
ERROR neutron.agent.l3.router_info self._process_internal_ports()
ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 656, in _process_internal_ports
ERROR neutron.agent.l3.router_info interface_name, ip_cidrs, p['mtu'])
ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/l3/ha_router.py", line 367, in internal_network_updated
ERROR neutron.agent.l3.router_info self._disable_ipv6_addressing_on_interface(interface_name)
ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/l3/ha_router.py", line 331, in _disable_ipv6_addressing_on_interface
ERROR neutron.agent.l3.router_info ipv6_lladdr = ip_lib.get_ipv6_lladdr(device.link.address)
ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 476, in address
ERROR neutron.agent.l3.router_info return self.attributes.get('link/ether')
ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 509, in attributes
ERROR neutron.agent.l3.router_info self._parent.namespace)
ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 247, in _wrap
ERROR neutron.agent.l3.router_info return self.channel.remote_call(name, args, kwargs)
ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 224, in remote_call
ERROR neutron.agent.l3.router_info raise exc_type(*result[2])
ERROR neutron.agent.l3.router_info neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-e05ab302-1b not found in namespace snat-9d5352e8-4f40-49a0-a[..truncated..]
ERROR neutron.agent.l3.router_info

Note that I am running OpenStack Victoria

[Slawek Kaplonski (slaweq)]

I have env for that so I will try to reproduce and triage it at least.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/799226

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/799439

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/799226
Committed: https://opendev.org/openstack/neutron/commit/b5dd6efdca0aa6e7405a55d66c7042a49ec72214
Submitter: "Zuul (22348)"
Branch: master

commit b5dd6efdca0aa6e7405a55d66c7042a49ec72214
Author: Slawek Kaplonski <email address hidden>
Date: Fri Jul 2 17:55:26 2021 +0200

    [DVR] Fix update of the MTU in the SNAT namespace

    When network's MTU is changed, Neutron sends notification about it
    to the L3 agents. In case of DVR (and DVR HA) MTU is then changed in
    the qrouter- namespace but should be also changed on snat interfaces
    in the snat namespace. And that part was missing.

    This patch adds special implementation of the internal_network_updated()
    method in the DvrEdgeRouter class so it can configure MTU also for
    in the snat namespace.

    This patch also removed passing attributes "interface_name",
    "ip_cidrs" and "mtu" to the internal_network_updated() method and adds
    "port" dict to be passed there. It is consistent with what is already
    done in e.g. internal_network_added() method and "port" dict is actually
    necessary to configure properly snat internal interface in the snat
    namespace.

    This patch adds also functional test of update network mtu for all types
    of routers as there was no such test at all.

    There is additional issue with DVR-HA which isn't fixed with that patch
    and for which follow up will be proposed. Because of that this patch is
    marked as partial fix for the related bug.

    Related-Bug: #1933273
    Change-Id: I200acfcaaae7f056ea9a563fead9ff2de8464971

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/812939

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/812950

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/812951

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/812952

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/812939
Committed: https://opendev.org/openstack/neutron/commit/67c9a305df31b9abc55c364e669e6b4cefb83a72
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 67c9a305df31b9abc55c364e669e6b4cefb83a72
Author: Slawek Kaplonski <email address hidden>
Date: Fri Jul 2 17:55:26 2021 +0200

    [DVR] Fix update of the MTU in the SNAT namespace

    When network's MTU is changed, Neutron sends notification about it
    to the L3 agents. In case of DVR (and DVR HA) MTU is then changed in
    the qrouter- namespace but should be also changed on snat interfaces
    in the snat namespace. And that part was missing.

    This patch adds special implementation of the internal_network_updated()
    method in the DvrEdgeRouter class so it can configure MTU also for
    in the snat namespace.

    This patch also removed passing attributes "interface_name",
    "ip_cidrs" and "mtu" to the internal_network_updated() method and adds
    "port" dict to be passed there. It is consistent with what is already
    done in e.g. internal_network_added() method and "port" dict is actually
    necessary to configure properly snat internal interface in the snat
    namespace.

    This patch adds also functional test of update network mtu for all types
    of routers as there was no such test at all.

    There is additional issue with DVR-HA which isn't fixed with that patch
    and for which follow up will be proposed. Because of that this patch is
    marked as partial fix for the related bug.

    Conflicts:
        neutron/tests/functional/agent/l3/test_dvr_router.py

    Related-Bug: #1933273
    Change-Id: I200acfcaaae7f056ea9a563fead9ff2de8464971
    (cherry picked from commit b5dd6efdca0aa6e7405a55d66c7042a49ec72214)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/812950
Committed: https://opendev.org/openstack/neutron/commit/763a65d9637c955a828edceec416c6c58744ff50
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 763a65d9637c955a828edceec416c6c58744ff50
Author: Slawek Kaplonski <email address hidden>
Date: Fri Jul 2 17:55:26 2021 +0200

    [DVR] Fix update of the MTU in the SNAT namespace

    When network's MTU is changed, Neutron sends notification about it
    to the L3 agents. In case of DVR (and DVR HA) MTU is then changed in
    the qrouter- namespace but should be also changed on snat interfaces
    in the snat namespace. And that part was missing.

    This patch adds special implementation of the internal_network_updated()
    method in the DvrEdgeRouter class so it can configure MTU also for
    in the snat namespace.

    This patch also removed passing attributes "interface_name",
    "ip_cidrs" and "mtu" to the internal_network_updated() method and adds
    "port" dict to be passed there. It is consistent with what is already
    done in e.g. internal_network_added() method and "port" dict is actually
    necessary to configure properly snat internal interface in the snat
    namespace.

    This patch adds also functional test of update network mtu for all types
    of routers as there was no such test at all.

    There is additional issue with DVR-HA which isn't fixed with that patch
    and for which follow up will be proposed. Because of that this patch is
    marked as partial fix for the related bug.

    Conflicts:
        neutron/tests/functional/agent/l3/test_dvr_router.py

    Related-Bug: #1933273
    Change-Id: I200acfcaaae7f056ea9a563fead9ff2de8464971
    (cherry picked from commit b5dd6efdca0aa6e7405a55d66c7042a49ec72214)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/812951
Committed: https://opendev.org/openstack/neutron/commit/384f2bb2aa85e33daad8b4fe0952fd18b96f10b8
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 384f2bb2aa85e33daad8b4fe0952fd18b96f10b8
Author: Slawek Kaplonski <email address hidden>
Date: Fri Jul 2 17:55:26 2021 +0200

    [DVR] Fix update of the MTU in the SNAT namespace

    When network's MTU is changed, Neutron sends notification about it
    to the L3 agents. In case of DVR (and DVR HA) MTU is then changed in
    the qrouter- namespace but should be also changed on snat interfaces
    in the snat namespace. And that part was missing.

    This patch adds special implementation of the internal_network_updated()
    method in the DvrEdgeRouter class so it can configure MTU also for
    in the snat namespace.

    This patch also removed passing attributes "interface_name",
    "ip_cidrs" and "mtu" to the internal_network_updated() method and adds
    "port" dict to be passed there. It is consistent with what is already
    done in e.g. internal_network_added() method and "port" dict is actually
    necessary to configure properly snat internal interface in the snat
    namespace.

    This patch adds also functional test of update network mtu for all types
    of routers as there was no such test at all.

    There is additional issue with DVR-HA which isn't fixed with that patch
    and for which follow up will be proposed. Because of that this patch is
    marked as partial fix for the related bug.

    Conflicts:
        neutron/tests/functional/agent/l3/test_dvr_router.py

    Related-Bug: #1933273
    Change-Id: I200acfcaaae7f056ea9a563fead9ff2de8464971
    (cherry picked from commit b5dd6efdca0aa6e7405a55d66c7042a49ec72214)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/799439
Committed: https://opendev.org/openstack/neutron/commit/21eabbcf03cb29be9899adbd6eaea207892cf07b
Submitter: "Zuul (22348)"
Branch: master

commit 21eabbcf03cb29be9899adbd6eaea207892cf07b
Author: Slawek Kaplonski <email address hidden>
Date: Mon Jul 5 09:40:28 2021 +0200

    [DVR] Fix update of the MTU in the DVR HA routers

    This is follow up of the patch [1] which fixed updating MTU in the
    snat namespace for the DVR routers.
    In case of DVR-HA routers there was additional issue with that as
    L3 agent tried to update MTU for the qr- interface in the
    self.ha_namespace which, for DVR-HA routers is snat namespace.

    This patch fixes that issue by setting MTU on the qr- interface in
    qrouter namespace and also setting MTU on the snat interface in snat
    namespace.

    [1] https://review.opendev.org/c/openstack/neutron/+/799226

    Closes-bug: #1933273
    Change-Id: I409bc674b65e4f495ebd42d03e97a09d51482339

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/816658

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/816659

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/816660

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/816661

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/816662

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816658
Committed: https://opendev.org/openstack/neutron/commit/7b2b6959716f95748bdfba1479f28b695cb80dab
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 7b2b6959716f95748bdfba1479f28b695cb80dab
Author: Slawek Kaplonski <email address hidden>
Date: Mon Jul 5 09:40:28 2021 +0200

    [DVR] Fix update of the MTU in the DVR HA routers

    This is follow up of the patch [1] which fixed updating MTU in the
    snat namespace for the DVR routers.
    In case of DVR-HA routers there was additional issue with that as
    L3 agent tried to update MTU for the qr- interface in the
    self.ha_namespace which, for DVR-HA routers is snat namespace.

    This patch fixes that issue by setting MTU on the qr- interface in
    qrouter namespace and also setting MTU on the snat interface in snat
    namespace.

    [1] https://review.opendev.org/c/openstack/neutron/+/799226

    Closes-bug: #1933273
    Change-Id: I409bc674b65e4f495ebd42d03e97a09d51482339
    (cherry picked from commit 21eabbcf03cb29be9899adbd6eaea207892cf07b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816659
Committed: https://opendev.org/openstack/neutron/commit/3ad894e032b345ed62da03fe197bb4bbd265a038
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3ad894e032b345ed62da03fe197bb4bbd265a038
Author: Slawek Kaplonski <email address hidden>
Date: Mon Jul 5 09:40:28 2021 +0200

    [DVR] Fix update of the MTU in the DVR HA routers

    This is follow up of the patch [1] which fixed updating MTU in the
    snat namespace for the DVR routers.
    In case of DVR-HA routers there was additional issue with that as
    L3 agent tried to update MTU for the qr- interface in the
    self.ha_namespace which, for DVR-HA routers is snat namespace.

    This patch fixes that issue by setting MTU on the qr- interface in
    qrouter namespace and also setting MTU on the snat interface in snat
    namespace.

    [1] https://review.opendev.org/c/openstack/neutron/+/799226

    Closes-bug: #1933273
    Change-Id: I409bc674b65e4f495ebd42d03e97a09d51482339
    (cherry picked from commit 21eabbcf03cb29be9899adbd6eaea207892cf07b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816660
Committed: https://opendev.org/openstack/neutron/commit/40f2ec340c94988b8fc7c2e5e4128cfe5ae9cbdb
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 40f2ec340c94988b8fc7c2e5e4128cfe5ae9cbdb
Author: Slawek Kaplonski <email address hidden>
Date: Mon Jul 5 09:40:28 2021 +0200

    [DVR] Fix update of the MTU in the DVR HA routers

    This is follow up of the patch [1] which fixed updating MTU in the
    snat namespace for the DVR routers.
    In case of DVR-HA routers there was additional issue with that as
    L3 agent tried to update MTU for the qr- interface in the
    self.ha_namespace which, for DVR-HA routers is snat namespace.

    This patch fixes that issue by setting MTU on the qr- interface in
    qrouter namespace and also setting MTU on the snat interface in snat
    namespace.

    [1] https://review.opendev.org/c/openstack/neutron/+/799226

    Closes-bug: #1933273
    Change-Id: I409bc674b65e4f495ebd42d03e97a09d51482339
    (cherry picked from commit 21eabbcf03cb29be9899adbd6eaea207892cf07b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816661
Committed: https://opendev.org/openstack/neutron/commit/3c99c719d08242f1ced941407d19d09d9d5add1e
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 3c99c719d08242f1ced941407d19d09d9d5add1e
Author: Slawek Kaplonski <email address hidden>
Date: Mon Jul 5 09:40:28 2021 +0200

    [DVR] Fix update of the MTU in the DVR HA routers

    This is follow up of the patch [1] which fixed updating MTU in the
    snat namespace for the DVR routers.
    In case of DVR-HA routers there was additional issue with that as
    L3 agent tried to update MTU for the qr- interface in the
    self.ha_namespace which, for DVR-HA routers is snat namespace.

    This patch fixes that issue by setting MTU on the qr- interface in
    qrouter namespace and also setting MTU on the snat interface in snat
    namespace.

    [1] https://review.opendev.org/c/openstack/neutron/+/799226

    Conflicts:
        neutron/tests/functional/agent/l3/test_dvr_router.py

    Closes-bug: #1933273
    Change-Id: I409bc674b65e4f495ebd42d03e97a09d51482339
    (cherry picked from commit 21eabbcf03cb29be9899adbd6eaea207892cf07b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.4.2 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/812952
Committed: https://opendev.org/openstack/neutron/commit/852aabe8defe7fc3b32f3a3522f0040b7b2d318c
Submitter: "Zuul (22348)"
Branch: stable/train

commit 852aabe8defe7fc3b32f3a3522f0040b7b2d318c
Author: Slawek Kaplonski <email address hidden>
Date: Fri Jul 2 17:55:26 2021 +0200

    [DVR] Fix update of the MTU in the SNAT namespace

    When network's MTU is changed, Neutron sends notification about it
    to the L3 agents. In case of DVR (and DVR HA) MTU is then changed in
    the qrouter- namespace but should be also changed on snat interfaces
    in the snat namespace. And that part was missing.

    This patch adds special implementation of the internal_network_updated()
    method in the DvrEdgeRouter class so it can configure MTU also for
    in the snat namespace.

    This patch also removed passing attributes "interface_name",
    "ip_cidrs" and "mtu" to the internal_network_updated() method and adds
    "port" dict to be passed there. It is consistent with what is already
    done in e.g. internal_network_added() method and "port" dict is actually
    necessary to configure properly snat internal interface in the snat
    namespace.

    This patch adds also functional test of update network mtu for all types
    of routers as there was no such test at all.

    There is additional issue with DVR-HA which isn't fixed with that patch
    and for which follow up will be proposed. Because of that this patch is
    marked as partial fix for the related bug.

    Conflicts:
        neutron/tests/functional/agent/l3/test_dvr_router.py

    Related-Bug: #1933273
    Change-Id: I200acfcaaae7f056ea9a563fead9ff2de8464971
    (cherry picked from commit b5dd6efdca0aa6e7405a55d66c7042a49ec72214)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816662
Committed: https://opendev.org/openstack/neutron/commit/262cef50a410f7987d49897a72d64059e25c369f
Submitter: "Zuul (22348)"
Branch: stable/train

commit 262cef50a410f7987d49897a72d64059e25c369f
Author: Slawek Kaplonski <email address hidden>
Date: Mon Jul 5 09:40:28 2021 +0200

    [DVR] Fix update of the MTU in the DVR HA routers

    This is follow up of the patch [1] which fixed updating MTU in the
    snat namespace for the DVR routers.
    In case of DVR-HA routers there was additional issue with that as
    L3 agent tried to update MTU for the qr- interface in the
    self.ha_namespace which, for DVR-HA routers is snat namespace.

    This patch fixes that issue by setting MTU on the qr- interface in
    qrouter namespace and also setting MTU on the snat interface in snat
    namespace.

    [1] https://review.opendev.org/c/openstack/neutron/+/799226

    Conflicts:
        neutron/tests/functional/agent/l3/test_dvr_router.py

    Closes-bug: #1933273
    Change-Id: I409bc674b65e4f495ebd42d03e97a09d51482339
    (cherry picked from commit 21eabbcf03cb29be9899adbd6eaea207892cf07b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/886408

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/886409

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.