I tried without setting advertisement_method_tenant_networks and explicitly setting it to host. Here is the log sequence from my attachment of my router to external gateway (--disable-snat).
2024-05-14 02:52:48.558 7 DEBUG ovn_bgp_agent.drivers.openstack.nb_ovn_bgp_driver [-] Adding BGP route for logical port with ip ['10.196.3.74'] _expose_ip /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/nb_ovn_bgp_driver.py:410
2024-05-14 02:52:48.569 7 DEBUG ovn_bgp_agent.utils.linux_net [-] Creating route at table 53: {'dst': '10.196.3.74', 'dst_len': 32, 'oif': 12, 'table': 53, 'proto': 3, 'scope': 253} add_ip_route /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/utils/linux_net.py:696
2024-05-14 02:52:48.570 7 DEBUG ovn_bgp_agent.utils.linux_net [-] Route created at table 53: {'dst': '10.196.3.74', 'dst_len': 32, 'oif': 12, 'table': 53, 'proto': 3, 'scope': 253} add_ip_route /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/utils/linux_net.py:698
2024-05-14 02:52:48.609 7 DEBUG ovn_bgp_agent.drivers.openstack.nb_ovn_bgp_driver [-] Added BGP route for logical port with ip ['10.196.3.74'] _expose_ip /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/nb_ovn_bgp_driver.py:443
After this, I captured the output from attaching a subnet to the router.
This behavior also occurs if I use subnet as the method. The only difference is that the 10.197.0.50 and 10.197.0.74 are not on the bgp-nic interface.
The route does appear on br-ex (in our case table 53).
10.197.0.0/25 via 10.196.3.74 dev br-ex
The router gateway appears on bgp-nic, but it is not reachable.
inet 10.196.3.74/32 scope global bgp-nic
valid_lft forever preferred_lft forever
Even if I add an address for 10.197.0.1/25 or 10.197.0.50/32 onto bgp-nic, I still do not have reachability.
It seems the NB driver for exposing tenant networks is broken.
On the flip side, if we use the ovn_bgp_driver (sb) - We see the VMs exposed and reachable; however, in order to reach additional IP addresses on the VMs (via allowed address pair) we have to add 10.197.0.1/25 to bgp-nic on the network node that has the router.
I tried without setting advertisement_ method_ tenant_ networks and explicitly setting it to host. Here is the log sequence from my attachment of my router to external gateway (--disable-snat).
2024-05-14 02:52:48.558 7 DEBUG ovn_bgp_ agent.drivers. openstack. nb_ovn_ bgp_driver [-] Adding BGP route for logical port with ip ['10.196.3.74'] _expose_ip /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/drivers/ openstack/ nb_ovn_ bgp_driver. py:410 agent.utils. linux_net [-] Creating route at table 53: {'dst': '10.196.3.74', 'dst_len': 32, 'oif': 12, 'table': 53, 'proto': 3, 'scope': 253} add_ip_route /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/utils/ linux_net. py:696 agent.utils. linux_net [-] Route created at table 53: {'dst': '10.196.3.74', 'dst_len': 32, 'oif': 12, 'table': 53, 'proto': 3, 'scope': 253} add_ip_route /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/utils/ linux_net. py:698 agent.drivers. openstack. nb_ovn_ bgp_driver [-] Added BGP route for logical port with ip ['10.196.3.74'] _expose_ip /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/drivers/ openstack/ nb_ovn_ bgp_driver. py:443
2024-05-14 02:52:48.569 7 DEBUG ovn_bgp_
2024-05-14 02:52:48.570 7 DEBUG ovn_bgp_
2024-05-14 02:52:48.609 7 DEBUG ovn_bgp_
After this, I captured the output from attaching a subnet to the router.
2024-05-14 02:54:33.677 7 DEBUG ovsdbapp. backend. ovs_idl. event [-] Matched UPDATE: LogicalSwitchPo rtSubnetAttachE vent(events= ('update' ,), table=' Logical_ Switch_ Port', conditions=None, old_conditions= None), priority=20 to row=Logical_ Switch_ Port(port_ security= [], addresses= ['router' ], type=router, dhcpv4_options=[], name=4ffd7337- 0593-42c3- b0f9-b40e75864e 76, up=[True], options= {'router- port': 'lrp-4ffd7337- 0593-42c3- b0f9-b40e75864e 76'}, ha_chassis_ group=[ ], external_ ids={'neutron: cidrs': '10.197.0.1/25', 'neutron: device_ id': '120eb8f3- 9575-4454- b975-aa9e416729 aa', 'neutron: device_ owner': 'network: router_ interface' , 'neutron:mtu': '', 'neutron: network_ name': 'neutron- 59b9bdf5- 0bcc-40e4- 8f29-31eb8db4aa 39', 'neutron: port_capabiliti es': '', 'neutron: port_name' : '', 'neutron: project_ id': '61bb60cb823e4a 7987b48dc29ba70 cd4', 'neutron: revision_ number' : '1', 'neutron: security_ group_ids' : '', 'neutron: subnet_ pool_addr_ scope4' : 'eb5af6f5- 2b92-4c89- 87ca-7c7964ca70 bd', 'neutron: subnet_ pool_addr_ scope6' : '', 'neutron: vnic_type' : 'normal'}, dynamic_ addresses= [], tag=[], parent_name=[], mirror_rules=[], tag_request=[], enabled=[True], dhcpv6_options=[]) old=Logical_ Switch_ Port(up= [False] ) matches /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovsdbapp/ backend/ ovs_idl/ event.py: 43 y.lockutils [-] Acquiring lock "nbbgp" by "ovn_bgp_ agent.drivers. openstack. nb_ovn_ bgp_driver. NBOVNBGPDriver. expose_ subnet" inner /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ oslo_concurrenc y/lockutils. py:404 y.lockutils [-] Lock "nbbgp" acquired by "ovn_bgp_ agent.drivers. openstack. nb_ovn_ bgp_driver. NBOVNBGPDriver. expose_ subnet" :: waited 0.000s inner /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ oslo_concurrenc y/lockutils. py:409 agent.drivers. openstack. utils.wire [-] Adding IP Rules for network 10.197.0.1/25 _wire_lrp_ port_underlay /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/drivers/ openstack/ utils/wire. py:597 agent.drivers. openstack. utils.wire [-] Added IP Rules for network 10.197.0.1/25 _wire_lrp_ port_underlay /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/drivers/ openstack/ utils/wire. py:604 agent.drivers. openstack. utils.wire [-] Adding IP Routes for network 10.197.0.1/25 _wire_lrp_ port_underlay /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/drivers/ openstack/ utils/wire. py:606 agent.utils. linux_net [-] Creating route at table 53: {'dst': '10.197.0.0', 'dst_len': 25, 'oif': 12, 'table': 53, 'proto': 3, 'gateway': '10.196.3.74', 'scope': 0} add_ip_route /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/utils/ linux_net. py:696 agent.utils. linux_net [-] Route created at table 53: {'dst': '10.197.0.0', 'dst_len': 25, 'oif': 12, 'table': 53, 'proto': 3, 'gateway': '10.196.3.74', 'scope': 0} add_ip_route /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/utils/ linux_net. py:698 agent.drivers. openstack. utils.wire [-] Added IP Routes for network 10.197.0.1/25 _wire_lrp_ port_underlay /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/drivers/ openstack/ utils/wire. py:621 agent.drivers. openstack. nb_ovn_ bgp_driver [-] Adding BGP route for tenant IP(s) ['10.197.0.74'] on chassis w2cosl0011104. heb.com _expose_remote_ip /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/drivers/ openstack/ nb_ovn_ bgp_driver. py:639 agent.drivers. openstack. nb_ovn_ bgp_driver [-] Added BGP route for tenant IP(s) ['10.197.0.74'] on chassis w2cosl0011104. heb.com _expose_remote_ip /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/drivers/ openstack/ nb_ovn_ bgp_driver. py:645 agent.drivers. openstack. nb_ovn_ bgp_driver [-] Adding BGP route for tenant IP(s) ['10.197.0.50'] on chassis w2cosl0011104. heb.com _expose_remote_ip /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/drivers/ openstack/ nb_ovn_ bgp_driver. py:639 agent.drivers. openstack. nb_ovn_ bgp_driver [-] Added BGP route for tenant IP(s) ['10.197.0.50'] on chassis w2cosl0011104. heb.com _expose_remote_ip /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ ovn_bgp_ agent/drivers/ openstack/ nb_ovn_ bgp_driver. py:645 y.lockutils [-] Lock "nbbgp" "released" by "ovn_bgp_ agent.drivers. openstack. nb_ovn_ bgp_driver. NBOVNBGPDriver. expose_ subnet" :: held 0.041s inner /var/lib/ kolla/venv/ lib64/python3. 9/site- packages/ oslo_concurrenc y/lockutils. py:423
2024-05-14 02:54:33.678 7 DEBUG oslo_concurrenc
2024-05-14 02:54:33.678 7 DEBUG oslo_concurrenc
2024-05-14 02:54:33.679 7 DEBUG ovn_bgp_
2024-05-14 02:54:33.680 7 DEBUG ovn_bgp_
2024-05-14 02:54:33.680 7 DEBUG ovn_bgp_
2024-05-14 02:54:33.692 7 DEBUG ovn_bgp_
2024-05-14 02:54:33.692 7 DEBUG ovn_bgp_
2024-05-14 02:54:33.693 7 DEBUG ovn_bgp_
2024-05-14 02:54:33.716 7 DEBUG ovn_bgp_
2024-05-14 02:54:33.718 7 DEBUG ovn_bgp_
2024-05-14 02:54:33.718 7 DEBUG ovn_bgp_
2024-05-14 02:54:33.720 7 DEBUG ovn_bgp_
2024-05-14 02:54:33.720 7 DEBUG oslo_concurrenc
The routing table from br-ex.
# ip route show table 53
default dev br-ex scope link
10.196.3.74 dev br-ex scope link
10.197.0.0/25 via 10.196.3.74 dev br-ex
Address list from bgp-nic:
# ip addr show bgp-nic NOARP,UP, LOWER_UP> mtu 1500 qdisc noqueue master bgp-vrf state UNKNOWN group default qlen 1000 8eff:fe33: 76b4/64 scope link
1864: bgp-nic: <BROADCAST,
link/ether fe:d5:8e:33:76:b4 brd ff:ff:ff:ff:ff:ff
inet 10.196.3.74/32 scope global bgp-nic
valid_lft forever preferred_lft forever
inet 10.197.0.74/32 scope global bgp-nic
valid_lft forever preferred_lft forever
inet 10.197.0.50/32 scope global bgp-nic
valid_lft forever preferred_lft forever
inet6 fe80::fcd5:
valid_lft forever preferred_lft forever
# ping 10.197.0.74
PING 10.197.0.74 (10.197.0.74) 56(84) bytes of data.
From 169.254.0.1 icmp_seq=1 Destination Host Unreachable
bgp-agent.conf for ovn-bgp-agent
[DEFAULT] kolla/openvswit ch/ovn- bgp-agent. log run/openvswitch /db.sock tenant_ networks = True 2b92-4c89- 87ca-7c7964ca70 bd snat_disabled_ for_tenant_ networks = True
debug = True
log_file = /var/log/
ovsdb_connection = unix:/var/
driver = nb_ovn_bgp_driver
exposing_method = underlay
bgp_AS = 64593
bgp_router_id = 99.98.1.3
evpn_nic = br-ex
evpn_local_ip = 99.98.1.3
expose_
address_scopes = eb5af6f5-
require_
I observe the following routes in the environment:
B>* 10.197.0.50/32 [20/0] is directly connected, bgp-nic (vrf bgp-vrf), weight 1, 00:04:29
B>* 10.197.0.74/32 [20/0] is directly connected, bgp-nic (vrf bgp-vrf), weight 1, 00:04:29
The IPs are not reachable at the moment.
This behavior also occurs if I use subnet as the method. The only difference is that the 10.197.0.50 and 10.197.0.74 are not on the bgp-nic interface.
The route does appear on br-ex (in our case table 53).
10.197.0.0/25 via 10.196.3.74 dev br-ex
The router gateway appears on bgp-nic, but it is not reachable.
inet 10.196.3.74/32 scope global bgp-nic
valid_lft forever preferred_lft forever
Even if I add an address for 10.197.0.1/25 or 10.197.0.50/32 onto bgp-nic, I still do not have reachability.
It seems the NB driver for exposing tenant networks is broken.
On the flip side, if we use the ovn_bgp_driver (sb) - We see the VMs exposed and reachable; however, in order to reach additional IP addresses on the VMs (via allowed address pair) we have to add 10.197.0.1/25 to bgp-nic on the network node that has the router.