Cannot setup IPv6 overlay network in xena

Bug #1952897 reported by Tamas Gergely Peter
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Tamas Gergely Peter

Bug Description

After upgrading from pike to xena my IPv6 vxlan overlay networks with linuxbridge stopped working.

In neutron-linuxbridge-agent.log on the compute nodes I see:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 465, in daemon_loop
   sync = self.process_network_devices(device_info)
  File "/usr/local/lib/python3.8/dist-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 214, in process_network_devices
    resync_a = self.treat_devices_added_updated(devices_added_updated)
  File "/usr/local/lib/python3.8/dist-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 231, in treat_devices_added_updated
    self._process_device_if_exists(device_details)
  File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 256, in _process_device_if_exists
    interface_plugged = self.mgr.plug_interface(
  File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 582, in plug_interface
    return self.add_tap_interface(network_id, network_segment.network_type,
  File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 521, in add_tap_interface
    return False
  File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 511, in add_tap_interface
    return self._add_tap_interface(network_id, network_type,
  File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 542, in _add_tap_interface
    elif not self.ensure_physical_in_bridge(network_id,
  File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 485, in ensure_physical_in_bridge
    return self.ensure_vxlan_bridge(network_id, segmentation_id, mtu)
  File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 260, in ensure_vxlan_bridge
    interface = self.ensure_vxlan(segmentation_id, mtu)
  File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 356, in ensure_vxlan
    int_vxlan = self.ip.add_vxlan(interface, segmentation_id,
  File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 303, in add_vxlan
    privileged.create_interface(name, self.namespace, "vxlan", **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 271, in _wrap
    return self.channel.remote_call(name, args, kwargs,
  File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 216, in remote_call
    raise exc_type(*result[2])
OSError: illegal IP address string passed to inet_pton

After some debugging I've found that xena uses pyroute2 module to create interfaces and it tries to parse the IPv6 vxlan_group and vxlan_local addresses with AF_INET address family, despite that the ml2_conf.ini contains "overlay_ip_version = 6".

If I apply the following modification to ip_lib.py, the issue disappears, however I know that this is just working until I use IPv6 vxlan networks.

--- ip_lib.py.orig 2021-11-30 16:12:31.434975862 +0000
+++ ip_lib.py 2021-12-01 10:44:20.737971722 +0000
@@ -280,7 +280,7 @@
                   local=None, srcport=None, dstport=None, proxy=False):
         kwargs = {'vxlan_id': vni}
         if group:
- kwargs['vxlan_group'] = group
+ kwargs['vxlan_group6'] = group
         if dev:
             kwargs['physical_interface'] = dev
         if ttl:
@@ -288,7 +288,7 @@
         if tos:
             kwargs['vxlan_tos'] = tos
         if local:
- kwargs['vxlan_local'] = local
+ kwargs['vxlan_local6'] = local
         if proxy:
             kwargs['vxlan_proxy'] = proxy
         # tuple: min,max

Is there any solution to use IPv6 overlay networks without issues?

tags: added: ipv6 linuxbridge
Revision history for this message
Lajos Katona (lajos-katona) wrote :

Hi, thanks for the bug report.
After some digging I found where pyroute2 ip.link for VXLAN have all possible params:
https://github.com/svinota/pyroute2/blob/master/pyroute2.core/pr2modules/netlink/rtnl/ifinfmsg/plugins/vxlan.py

I am not sure which pyroute2 is installed for your xena env.

Perhaps the simplest way is similar to your suggestion, but check ip family from local, and call privsep.create_interface with the correct (ipv4 or ipv6) kwargs.

If you would like to and have time please propose the patch (to master, we can backport from master only) or I can propose as well.

Changed in neutron:
status: New → Confirmed
importance: Undecided → High
Changed in neutron:
assignee: nobody → Tamas Gergely Peter (tamasgp)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/820376

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/820376
Committed: https://opendev.org/openstack/neutron/commit/a47e9494c1d8895425ad3a05a2c2290c74f0b647
Submitter: "Zuul (22348)"
Branch: master

commit a47e9494c1d8895425ad3a05a2c2290c74f0b647
Author: Tamas Gergely Peter <email address hidden>
Date: Tue Dec 28 16:00:36 2021 +0000

    Check whether vxlan group and local addresses are IPv4 or IPv6

    Check if group and/or local addresses passed to ip_lib / add_vxlan()
    are IPv4 or IPv6. In case of IPv4 fill 'vxlan_group' and 'vxlan_local'
    arguments and in case of IPv6 fill 'vxlan_group6' and 'vxlan_local6'
    arguments to be passed down to privileged create_interface() method.
    In case of an invalid address format raise an AddrFormatError exception.

    Closes-Bug: #1952897
    Change-Id: I2e3b0c1635627edb2c86c6120b0410ab3c4678b2

Changed in neutron:
status: In Progress → Fix Released
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.0.0.0rc1

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/833202

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/833203

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/833373

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/833204

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/833205

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/833202
Committed: https://opendev.org/openstack/neutron/commit/c6d3f90bee85e7c18ee756d1a9e837e93e0400a6
Submitter: "Zuul (22348)"
Branch: stable/xena

commit c6d3f90bee85e7c18ee756d1a9e837e93e0400a6
Author: Tamas Gergely Peter <email address hidden>
Date: Tue Dec 28 16:00:36 2021 +0000

    Check whether vxlan group and local addresses are IPv4 or IPv6

    Check if group and/or local addresses passed to ip_lib / add_vxlan()
    are IPv4 or IPv6. In case of IPv4 fill 'vxlan_group' and 'vxlan_local'
    arguments and in case of IPv6 fill 'vxlan_group6' and 'vxlan_local6'
    arguments to be passed down to privileged create_interface() method.
    In case of an invalid address format raise an AddrFormatError exception.

    Closes-Bug: #1952897
    Change-Id: I2e3b0c1635627edb2c86c6120b0410ab3c4678b2
    (cherry picked from commit a47e9494c1d8895425ad3a05a2c2290c74f0b647)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/833203
Committed: https://opendev.org/openstack/neutron/commit/a159a0d9bc3fff06a8b588f52e370f32d82a6b8b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit a159a0d9bc3fff06a8b588f52e370f32d82a6b8b
Author: Tamas Gergely Peter <email address hidden>
Date: Tue Dec 28 16:00:36 2021 +0000

    Check whether vxlan group and local addresses are IPv4 or IPv6

    Check if group and/or local addresses passed to ip_lib / add_vxlan()
    are IPv4 or IPv6. In case of IPv4 fill 'vxlan_group' and 'vxlan_local'
    arguments and in case of IPv6 fill 'vxlan_group6' and 'vxlan_local6'
    arguments to be passed down to privileged create_interface() method.
    In case of an invalid address format raise an AddrFormatError exception.

    Conflicts:
        neutron/agent/linux/ip_lib.py
        neutron/tests/unit/agent/linux/test_ip_lib.py

    Closes-Bug: #1952897
    Change-Id: I2e3b0c1635627edb2c86c6120b0410ab3c4678b2
    (cherry picked from commit a47e9494c1d8895425ad3a05a2c2290c74f0b647)

tags: added: in-stable-wallaby
tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/833373
Committed: https://opendev.org/openstack/neutron/commit/4958720448a267b5eabfb36e2e4a28c3ab75cf2e
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 4958720448a267b5eabfb36e2e4a28c3ab75cf2e
Author: Tamas Gergely Peter <email address hidden>
Date: Tue Dec 28 16:00:36 2021 +0000

    Check whether vxlan group and local addresses are IPv4 or IPv6

    Check if group and/or local addresses passed to ip_lib / add_vxlan()
    are IPv4 or IPv6. In case of IPv4 fill 'vxlan_group' and 'vxlan_local'
    arguments and in case of IPv6 fill 'vxlan_group6' and 'vxlan_local6'
    arguments to be passed down to privileged create_interface() method.
    In case of an invalid address format raise an AddrFormatError exception.

    Conflicts:
        neutron/agent/linux/ip_lib.py
        neutron/tests/functional/agent/linux/test_ip_lib.py
        neutron/tests/unit/agent/linux/test_ip_lib.py

    Closes-Bug: #1952897
    Change-Id: I2e3b0c1635627edb2c86c6120b0410ab3c4678b2
    (cherry picked from commit a47e9494c1d8895425ad3a05a2c2290c74f0b647)
    (cherry picked from commit 45297dd84a8e118e0a4daff8d5e8d6bfc10f9e80)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/833204
Committed: https://opendev.org/openstack/neutron/commit/efb2ab8cc0bd1b6a2d01cb0e854117d191665074
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit efb2ab8cc0bd1b6a2d01cb0e854117d191665074
Author: Tamas Gergely Peter <email address hidden>
Date: Tue Dec 28 16:00:36 2021 +0000

    Check whether vxlan group and local addresses are IPv4 or IPv6

    Check if group and/or local addresses passed to ip_lib / add_vxlan()
    are IPv4 or IPv6. In case of IPv4 fill 'vxlan_group' and 'vxlan_local'
    arguments and in case of IPv6 fill 'vxlan_group6' and 'vxlan_local6'
    arguments to be passed down to privileged create_interface() method.
    In case of an invalid address format raise an AddrFormatError exception.

    Conflicts:
        neutron/agent/linux/ip_lib.py
        neutron/tests/functional/agent/linux/test_ip_lib.py
        neutron/tests/unit/agent/linux/test_ip_lib.py

    Closes-Bug: #1952897
    Change-Id: I2e3b0c1635627edb2c86c6120b0410ab3c4678b2
    (cherry picked from commit a47e9494c1d8895425ad3a05a2c2290c74f0b647)
    (cherry picked from commit 45297dd84a8e118e0a4daff8d5e8d6bfc10f9e80)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/833205
Committed: https://opendev.org/openstack/neutron/commit/b17ece5e05183f33037466aa8d83ab11ba5b0f33
Submitter: "Zuul (22348)"
Branch: stable/train

commit b17ece5e05183f33037466aa8d83ab11ba5b0f33
Author: Tamas Gergely Peter <email address hidden>
Date: Tue Dec 28 16:00:36 2021 +0000

    Check whether vxlan group and local addresses are IPv4 or IPv6

    Check if group and/or local addresses passed to ip_lib / add_vxlan()
    are IPv4 or IPv6. In case of IPv4 fill 'vxlan_group' and 'vxlan_local'
    arguments and in case of IPv6 fill 'vxlan_group6' and 'vxlan_local6'
    arguments to be passed down to privileged create_interface() method.
    In case of an invalid address format raise an AddrFormatError exception.

    Conflicts:
        neutron/agent/linux/ip_lib.py
        neutron/tests/functional/agent/linux/test_ip_lib.py
        neutron/tests/unit/agent/linux/test_ip_lib.py

    Closes-Bug: #1952897
    Change-Id: I2e3b0c1635627edb2c86c6120b0410ab3c4678b2
    (cherry picked from commit a47e9494c1d8895425ad3a05a2c2290c74f0b647)
    (cherry picked from commit 45297dd84a8e118e0a4daff8d5e8d6bfc10f9e80)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.4.0

This issue was fixed in the openstack/neutron 17.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.3.0

This issue was fixed in the openstack/neutron 18.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.2.0

This issue was fixed in the openstack/neutron 19.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron train-eol

This issue was fixed in the openstack/neutron train-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron ussuri-eol

This issue was fixed in the openstack/neutron ussuri-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.