neutron

[OVN] Cannot create metadata port for segmented network

Bug #1871608 reported by Maciej Jozefczyk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Maciej Jozefczyk

Bug Description

While following instructions for Routed Provided Networks [1] and using OVN Neutron raises error during creation on second segment subnet:

==== How to reproduce ===

sudo iniset /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vlan network_vlan_ranges segment-1:100:102,segment-2:200:202
sudo systemctl restart devstack@q-svc

openstack network create --share --provider-physical-network segment-1 --provider-network-type vlan --provider-segment 100 public-multisegment
openstack network segment set --name segment-1 $(openstack network segment list --network public-multisegment -c ID -f value)
openstack network segment create --physical-network segment-2 --network-type vlan --segment 200 --network public-multisegment segment-2

openstack subnet create --network public-multisegment --network-segment segment-1 --ip-version 4 --subnet-range 172.24.4.0/24 --allocation-pool start=172.24.4.100,end=172.24.4.200 public-multisegment-segment-1-v4
openstack subnet create --network public-multisegment --network-segment segment-2 --ip-version 4 --subnet-range 172.24.6.0/24 --allocation-pool start=172.24.6.100,end=172.24.6.200 public-multisegment-segment-2-v4

EXCEPTION RAISED ON LAST COMMAND:
Apr 08 11:23:35 central neutron-server[10871]: DEBUG neutron_lib.callbacks.manager [None req-e975c78f-bb1d-449d-9517-0a9386733b13 demo admin] Notify callbacks ['neutron.plugins.ml2.plugin.SecurityGroupDbMixin._ensure_default_security_group
_handler--9223372036853431474'] for port, before_update {{(pid=10878) _notify_loop /usr/local/lib/python3.6/dist-packages/neutron_lib/callbacks/manager.py:193}}
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers [None req-e975c78f-bb1d-449d-9517-0a9386733b13 demo admin] Mechanism driver 'ovn' failed in create_subnet_postcommit: neutron.services.segments.exceptions.Fi
xedIpsSubnetsNotOnSameSegment: Cannot allocate addresses from different segments.
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/home/vagrant/neutron/neutron/plugins/ml2/managers.py", line 477, in _call_on_drivers
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/home/vagrant/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 441, in create_subnet_postcommit
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers context.network.current)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/home/vagrant/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2165, in create_subnet
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers if subnet['enable_dhcp']:
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/home/vagrant/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2356, in update_metadata_port
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers metadata_port['id'], port)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/home/vagrant/neutron/neutron/common/utils.py", line 685, in inner
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers return f(self, context, *args, **kwargs)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 233, in wrapped
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers return method(*args, **kwargs)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 139, in wrapped
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers setattr(e, '_RETRY_EXCEEDED', True)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers self.force_reraise()
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers six.reraise(self.type_, self.value, self.tb)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers raise value
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers return f(*args, **kwargs)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 154, in wrapper
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers ectxt.value = e.inner_exc
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers self.force_reraise()
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers six.reraise(self.type_, self.value, self.tb)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers raise value
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 142, in wrapper
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers return f(*args, **kwargs)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers LOG.debug("Retry wrapper got retriable exception: %s", e)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers self.force_reraise()
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers six.reraise(self.type_, self.value, self.tb)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers raise value
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 179, in wrapped Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers return f(*dup_args, **dup_kwargs)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/home/vagrant/neutron/neutron/plugins/ml2/plugin.py", line 1652, in update_port Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers port)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 233, in wrapped Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers return method(*args, **kwargs)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/home/vagrant/neutron/neutron/db/db_base_plugin_v2.py", line 1503, in update_port Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers new_port=new_port)
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/home/vagrant/neutron/neutron/db/ipam_backend_mixin.py", line 703, in update_port Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers new_port.get('mac_address'))
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/home/vagrant/neutron/neutron/db/ipam_pluggable_backend.py", line 428, in update_port_with_ips
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers new_mac) Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/home/vagrant/neutron/neutron/db/ipam_pluggable_backend.py", line 351, in _update_ips_for_port
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers fixed_ips=changes.add + changes.original) Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/home/vagrant/neutron/neutron/db/ipam_backend_mixin.py", line 657, in _ipam_get_subnets
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers fixed_ips) Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/home/vagrant/neutron/neutron/objects/subnet.py", line 338, in find_candidate_subnets
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers query, fixed_ips).all() Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers File "/home/vagrant/neutron/neutron/objects/subnet.py", line 394, in _query_filter_by_fixed_ips_segment
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers raise segment_exc.FixedIpsSubnetsNotOnSameSegment() Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers neutron.services.segments.exceptions.FixedIpsSubnetsNotOnSameSegment: Cannot allocate addresses from different segments.
Apr 08 11:23:35 central neutron-server[10871]: ERROR neutron.plugins.ml2.managers

Client error:
BadRequestException: 400: Client Error for url: http://192.168.121.158:9696/v2.0/subnets, Cannot allocate addresses from different segments.

Looks like we need to fix segment handling for metadata ports [2].

[1] https://docs.openstack.org/ocata/networking-guide/config-routed-networks.html
[2] https://github.com/openstack/neutron/blob/805bd1210e2286c02e59b2462ec6470ec81155b0/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L2127

Maciej Jozefczyk (maciejjozefczyk)
summary: - [OVN] Cannot create metadata part for segmented network
+ [OVN] Cannot create metadata port for segmented network
Revision history for this message
Daniel Alvarez (dalvarezs) wrote :

Can we override this behavior in the OVN driver to allow IP allocations from multiple segment if the port type is 'localport' ? That'd be awesome :)

Revision history for this message
Lajos Katona (lajos-katona) wrote :

In a fresh build from master neutron I have no such issue, do I miss something?

$ grep service_plugins /etc/neutron/neutron.conf
service_plugins = ovn-router,trunk,segments

$ grep vlan_ranges /etc/neutron/plugins/ml2/ml2_conf.ini
network_vlan_ranges = public,segment-1:100:102,segment-2:200:202

my local.conf is from here: https://opendev.org/openstack/neutron/src/branch/master/devstack/ovn-local.conf.sample
(as manual step I added segments to service_plugins list)

Revision history for this message
Maciej Jozefczyk (maciejjozefczyk) wrote :

Hey

you should be able to reproduce it. While setting a second subnet linked to segment-2 withing the same network OVN try to update localnet port, which handles metadata requests, to add new IP address from second subnet.

The problem started happening recently after merging this valid patch [1].
It ensures that while adding a new fixed_ip to the port, the port has no other fixed ip from different segment configured.
In our case its exactly why it fails, because we are explicitly trying to add second fixed_ip.

[1] https://review.opendev.org/#/c/709444/9/neutron/objects/subnet.py

Revision history for this message
Lajos Katona (lajos-katona) wrote :

Hi, confirmed, I had issues in my devstack VM with fetching and it was hidden in tons of logs

Changed in neutron:
status: New → Confirmed
importance: Undecided → High
Maciej Jozefczyk (maciejjozefczyk)
Changed in neutron:
assignee: nobody → Maciej Jozefczyk (maciej.jozefczyk)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/720464

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/720464
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8d1512afb07a9a77958c44f2e7f5d47865c8a475
Submitter: Zuul
Branch: master

commit 8d1512afb07a9a77958c44f2e7f5d47865c8a475
Author: Maciej Józefczyk <email address hidden>
Date: Thu Apr 16 12:58:15 2020 +0000

    [OVN] Allow IP allocation with different segments for OVN service ports

    OVN creates localport [1] for each network that has metadata
    and allocate IP address from subnet within this network that has
    DHCP enabled. The traffic from this port will never go outside
    the chassis.

    While using multiple segments with subnet linked to each segment
    OVN needs to create an allocation of IP address for each of those
    subnets [2] in order to generate data for OVN NBDB IPv4 DHCP Options.

    The change [3] started to validate that condition, while multiple
    IP addresses from different segments are tried to be allocated on
    one port. We can skip this for OVN Metadata port, because there
    is no reason to prevent those kind of allocation for OVN.

    [1] http://www.openvswitch.org/support/dist-docs/ovn-architecture.7.html
    [2] https://github.com/openstack/neutron/blob/5f42488a9a6a7a3afa55a56f53e2f5ef09365115/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L2279
    [3] https://review.opendev.org/#/c/709444/

    Change-Id: Ib51cde89ed873f48db4daebc27a0980da9cc0f19
    Closes-Bug: 1871608

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/742851

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/742872

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/742851
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ab32d7ae64fada57863565f2c924971a42c4145b
Submitter: Zuul
Branch: stable/ussuri

commit ab32d7ae64fada57863565f2c924971a42c4145b
Author: Maciej Józefczyk <email address hidden>
Date: Thu Apr 16 12:58:15 2020 +0000

    [OVN] Allow IP allocation with different segments for OVN service ports

    OVN creates localport [1] for each network that has metadata
    and allocate IP address from subnet within this network that has
    DHCP enabled. The traffic from this port will never go outside
    the chassis.

    While using multiple segments with subnet linked to each segment
    OVN needs to create an allocation of IP address for each of those
    subnets [2] in order to generate data for OVN NBDB IPv4 DHCP Options.

    The change [3] started to validate that condition, while multiple
    IP addresses from different segments are tried to be allocated on
    one port. We can skip this for OVN Metadata port, because there
    is no reason to prevent those kind of allocation for OVN.

    For stable branches we decide if a port is distributed or not
    by looking for DEVICE_OWNER_DHCP device owner and `ovn` device_id,
    instead DEVICE_OWNER_DISTRIBUTED device owner.

    Conflicts:
       neutron/db/ipam_backend_mixin.py
       neutron/tests/unit/db/test_ipam_pluggable_backend.py

    [1] http://www.openvswitch.org/support/dist-docs/ovn-architecture.7.html
    [2] https://github.com/openstack/neutron/blob/5f42488a9a6a7a3afa55a56f53e2f5ef09365115/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L2279
    [3] https://review.opendev.org/#/c/709444/

    Change-Id: Ib51cde89ed873f48db4daebc27a0980da9cc0f19
    Closes-Bug: 1871608
    (cherry picked from commit 8d1512afb07a9a77958c44f2e7f5d47865c8a475)

tags: added: in-stable-ussuri
tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/742872
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8f1912a1d195dc69eed9d9aa8ff8c4ef12448b75
Submitter: Zuul
Branch: stable/train

commit 8f1912a1d195dc69eed9d9aa8ff8c4ef12448b75
Author: Maciej Józefczyk <email address hidden>
Date: Thu Apr 16 12:58:15 2020 +0000

    [OVN] Allow IP allocation with different segments for OVN service ports

    OVN creates localport [1] for each network that has metadata
    and allocate IP address from subnet within this network that has
    DHCP enabled. The traffic from this port will never go outside
    the chassis.

    While using multiple segments with subnet linked to each segment
    OVN needs to create an allocation of IP address for each of those
    subnets [2] in order to generate data for OVN NBDB IPv4 DHCP Options.

    The change [3] started to validate that condition, while multiple
    IP addresses from different segments are tried to be allocated on
    one port. We can skip this for OVN Metadata port, because there
    is no reason to prevent those kind of allocation for OVN.

    For stable branches we decide if a port is distributed or not
    by looking for DEVICE_OWNER_DHCP device owner and `ovn` device_id,
    instead DEVICE_OWNER_DISTRIBUTED device owner.

    Conflicts:
       neutron/db/ipam_backend_mixin.py
       neutron/tests/unit/db/test_ipam_pluggable_backend.py
       neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py

    [1] http://www.openvswitch.org/support/dist-docs/ovn-architecture.7.html
    [2] https://github.com/openstack/neutron/blob/5f42488a9a6a7a3afa55a56f53e2f5ef09365115/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L2279
    [3] https://review.opendev.org/#/c/709444/

    Change-Id: Ib51cde89ed873f48db4daebc27a0980da9cc0f19
    Closes-Bug: 1871608
    (cherry picked from commit 8d1512afb07a9a77958c44f2e7f5d47865c8a475)

Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/745829
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7f551db9be861c531d913aa428dd8498e290eaaf
Submitter: Zuul
Branch: master

commit 7f551db9be861c531d913aa428dd8498e290eaaf
Author: Flavio Fernandes <email address hidden>
Date: Wed Aug 12 06:41:37 2020 -0400

    [OVN] Update sample local.conf to support segments

    This change updates sample OVN local.conf file in order
    to enable neutron-segments service.

    Related-Bug: #1871608
    Closes-Bug: #1891243
    Change-Id: Iac6227093b0e851716a76e38145508d6724c650a

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.