Cannot provision flat network after reconfiguring physical bridges

[Anton Kurbatov]

I ran into a problem when the network connectivity inside the newly created VM is not working.

* Pre-conditions:
- the neutron ovs agent has not yet seen any ports from the VM network;
- any other bridge (except for the network in which the VM is created) is recreated on the node.

* Step-by-step reproduction steps:

The bridge mapping from ml2_conf.ini looks like:
[ovs]
bridge_mappings = Public:br-eth0,Test:br-test

The 'Test:br-test' mapping is a test bridge to demonstrate the problem.
I've created it using ovs-vsctl tool like: ovs-vsctl add-br br-test.

1) Recreate this test bridge that triggers _reconfigure_physical_bridges:

[root@sqvm2-2009 ~]# ovs-vsctl del-br br-test; ovs-vsctl add-br br-test
[root@sqvm2-2009 ~]#

2) Create the first VM from the 'public' network that is mapped to the 'Public' bridge and try to ping it:
---
[root@sqvm2-2009 ~]# openstack server create test-vm --image cirros --flavor 100 --network public --boot-from-volume 1
[root@sqvm2-2009 ~]# openstack server list
+--------------------------------------+---------+--------+---------------------+--------------------------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+---------+--------+---------------------+--------------------------+--------+
| 68c32b4d-8f90-4ced-8ca4-67a9e4ff255b | test-vm | ACTIVE | public=10.34.111.12 | N/A (booted from volume) | tiny |
+--------------------------------------+---------+--------+---------------------+--------------------------+--------+
[root@sqvm2-2009 ~]# virsh console 68c32b4d-8f90-4ced-8ca4-67a9e4ff255b
Connected to domain instance-00000005
Escape character is ^]

login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
cirros login: cirros
Password:
$ sudo ip addr add 10.34.111.12/18 dev eth0
$ ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:67:f8:4e brd ff:ff:ff:ff:ff:ff
    inet 10.34.111.12/18 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe67:f84e/64 scope link
       valid_lft forever preferred_lft forever
$
[root@sqvm2-2009 ~]# ping 10.34.111.12
PING 10.34.111.12 (10.34.111.12) 56(84) bytes of data.
From 10.34.66.138 icmp_seq=1 Destination Host Unreachable
From 10.34.66.138 icmp_seq=2 Destination Host Unreachable
From 10.34.66.138 icmp_seq=3 Destination Host Unreachable
From 10.34.66.138 icmp_seq=4 Destination Host Unreachable
^C
--- 10.34.111.12 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4000ms
pipe 4
[root@sqvm2-2009 ~]#
---

* Actual result:
The VM is not pingable, but should be.
See the logs of port processing below:

2021-05-24 17:28:29.776 13744 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7f3667e0-56a5-4830-8376-10577a2ee167 - - - - -] Port c010955a-4782-418d-a612-0bfbd66b3c09 updated. Details: {'device': 'c010955a-4782-418d-a612-0bfbd66b3c09', 'device_id': '68c32b4d-8f90-4ced-8ca4-67a9e4ff255b', 'network_id': '568fb8ce-8f1b-456e-8a31-330ef19f2f5c', 'port_id': 'c010955a-4782-418d-a612-0bfbd66b3c09', 'mac_address': 'fa:16:3e:67:f8:4e', 'admin_state_up': True, 'network_type': 'flat', 'segmentation_id': None, 'physical_network': 'Public', 'fixed_ips': [{'subnet_id': 'b6f963e3-ad77-4bde-8431-049f87871422', 'ip_address': '10.34.111.12'}], 'device_owner': 'compute:nova', 'allowed_address_pairs': [], 'port_security_enabled': True, 'qos_policy_id': None, 'network_qos_policy_id': None, 'profile': {}, 'vif_type': 'ovs', 'vnic_type': 'normal', 'security_groups': ['09948793-2e11-4d89-ad1f-0c0d0eef80f0'], 'migrating_to': None}
2021-05-24 17:28:29.776 13744 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7f3667e0-56a5-4830-8376-10577a2ee167 - - - - -] Assigning 2 as local vlan for net-id=568fb8ce-8f1b-456e-8a31-330ef19f2f5c
2021-05-24 17:28:29.777 13744 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7f3667e0-56a5-4830-8376-10577a2ee167 - - - - -] Cannot provision flat network for net-id=568fb8ce-8f1b-456e-8a31-330ef19f2f5c - no bridge for physical_network Public

* Version:
Stein release.
The issue is also reproducible on the master branch.

* Attachments:
Full neutron-openvswitch-agent service logs attached

[Anton Kurbatov (akurbatov)]

I would like someone to confirm that this bug have place, may be Slawek Kaplonski as the author of
_reconfigure_physical_bridges in ovs agent.

I am new to contributing to openstack. Let me send the patch myself.
This patch fixes the issue:

diff --git a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
index 2c209bd..cfe0534 100644
--- a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
+++ b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
@@ -225,6 +225,9 @@ class OVSNeutronAgent(l2population_rpc.L2populationRpcCallBackTunnelMixin,
             {k: [v] for k, v in self.bridge_mappings.items()}
         )

+ self.phys_brs = {}
+ self.int_ofports = {}
+ self.phys_ofports = {}
         self.setup_physical_bridges(self.bridge_mappings)
         self.vlan_manager = vlanmanager.LocalVlanManager()

@@ -1485,9 +1488,6 @@ class OVSNeutronAgent(l2population_rpc.L2populationRpcCallBackTunnelMixin,

         :param bridge_mappings: map physical network names to bridge names.
         '''
- self.phys_brs = {}
- self.int_ofports = {}
- self.phys_ofports = {}
         datapath_ids_set = set()
         ovs = ovs_lib.BaseOVS()
         ovs_bridges = ovs.get_bridges()

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/793015

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/793015
Committed: https://opendev.org/openstack/neutron/commit/edc3852c83980d722a09eab17ca3c2214c79ee64
Submitter: "Zuul (22348)"
Branch: master

commit edc3852c83980d722a09eab17ca3c2214c79ee64
Author: Anton Kurbatov <email address hidden>
Date: Tue May 25 17:11:04 2021 +0000

    Fix phys network lost after reconfigure

    This patch moves phys_brs and 'self' associated attributes from
    setup_physical_bridges to __init__. Otherwise, bridges from bridge_mapping
    could be lost, e.g. when setup_physical_bridges is called with only
    reconfigured bridges from _do_reconfigure_physical_bridges.

    Closes-Bug: #1929438
    Signed-off-by: Anton Kurbatov <email address hidden>
    Change-Id: Ieadb801e48898e8b654153ad37be80dd9c865413

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.