Comment 11 for bug 2017886

I tried without setting advertisement_method_tenant_networks and explicitly setting it to host. Here is the log sequence from my attachment of my router to external gateway (--disable-snat).

2024-05-14 02:52:48.558 7 DEBUG ovn_bgp_agent.drivers.openstack.nb_ovn_bgp_driver [-] Adding BGP route for logical port with ip ['10.196.3.74'] _expose_ip /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/nb_ovn_bgp_driver.py:410
2024-05-14 02:52:48.569 7 DEBUG ovn_bgp_agent.utils.linux_net [-] Creating route at table 53: {'dst': '10.196.3.74', 'dst_len': 32, 'oif': 12, 'table': 53, 'proto': 3, 'scope': 253} add_ip_route /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/utils/linux_net.py:696
2024-05-14 02:52:48.570 7 DEBUG ovn_bgp_agent.utils.linux_net [-] Route created at table 53: {'dst': '10.196.3.74', 'dst_len': 32, 'oif': 12, 'table': 53, 'proto': 3, 'scope': 253} add_ip_route /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/utils/linux_net.py:698
2024-05-14 02:52:48.609 7 DEBUG ovn_bgp_agent.drivers.openstack.nb_ovn_bgp_driver [-] Added BGP route for logical port with ip ['10.196.3.74'] _expose_ip /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/nb_ovn_bgp_driver.py:443

After this, I captured the output from attaching a subnet to the router.

2024-05-14 02:54:33.677 7 DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched UPDATE: LogicalSwitchPortSubnetAttachEvent(events=('update',), table='Logical_Switch_Port', conditions=None, old_conditions=None), priority=20 to row=Logical_Switch_Port(port_security=[], addresses=['router'], type=router, dhcpv4_options=[], name=4ffd7337-0593-42c3-b0f9-b40e75864e76, up=[True], options={'router-port': 'lrp-4ffd7337-0593-42c3-b0f9-b40e75864e76'}, ha_chassis_group=[], external_ids={'neutron:cidrs': '10.197.0.1/25', 'neutron:device_id': '120eb8f3-9575-4454-b975-aa9e416729aa', 'neutron:device_owner': 'network:router_interface', 'neutron:mtu': '', 'neutron:network_name': 'neutron-59b9bdf5-0bcc-40e4-8f29-31eb8db4aa39', 'neutron:port_capabilities': '', 'neutron:port_name': '', 'neutron:project_id': '61bb60cb823e4a7987b48dc29ba70cd4', 'neutron:revision_number': '1', 'neutron:security_group_ids': '', 'neutron:subnet_pool_addr_scope4': 'eb5af6f5-2b92-4c89-87ca-7c7964ca70bd', 'neutron:subnet_pool_addr_scope6': '', 'neutron:vnic_type': 'normal'}, dynamic_addresses=[], tag=[], parent_name=[], mirror_rules=[], tag_request=[], enabled=[True], dhcpv6_options=[]) old=Logical_Switch_Port(up=[False]) matches /var/lib/kolla/venv/lib64/python3.9/site-packages/ovsdbapp/backend/ovs_idl/event.py:43
2024-05-14 02:54:33.678 7 DEBUG oslo_concurrency.lockutils [-] Acquiring lock "nbbgp" by "ovn_bgp_agent.drivers.openstack.nb_ovn_bgp_driver.NBOVNBGPDriver.expose_subnet" inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:404
2024-05-14 02:54:33.678 7 DEBUG oslo_concurrency.lockutils [-] Lock "nbbgp" acquired by "ovn_bgp_agent.drivers.openstack.nb_ovn_bgp_driver.NBOVNBGPDriver.expose_subnet" :: waited 0.000s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:409
2024-05-14 02:54:33.679 7 DEBUG ovn_bgp_agent.drivers.openstack.utils.wire [-] Adding IP Rules for network 10.197.0.1/25 _wire_lrp_port_underlay /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/utils/wire.py:597
2024-05-14 02:54:33.680 7 DEBUG ovn_bgp_agent.drivers.openstack.utils.wire [-] Added IP Rules for network 10.197.0.1/25 _wire_lrp_port_underlay /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/utils/wire.py:604
2024-05-14 02:54:33.680 7 DEBUG ovn_bgp_agent.drivers.openstack.utils.wire [-] Adding IP Routes for network 10.197.0.1/25 _wire_lrp_port_underlay /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/utils/wire.py:606
2024-05-14 02:54:33.692 7 DEBUG ovn_bgp_agent.utils.linux_net [-] Creating route at table 53: {'dst': '10.197.0.0', 'dst_len': 25, 'oif': 12, 'table': 53, 'proto': 3, 'gateway': '10.196.3.74', 'scope': 0} add_ip_route /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/utils/linux_net.py:696
2024-05-14 02:54:33.692 7 DEBUG ovn_bgp_agent.utils.linux_net [-] Route created at table 53: {'dst': '10.197.0.0', 'dst_len': 25, 'oif': 12, 'table': 53, 'proto': 3, 'gateway': '10.196.3.74', 'scope': 0} add_ip_route /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/utils/linux_net.py:698
2024-05-14 02:54:33.693 7 DEBUG ovn_bgp_agent.drivers.openstack.utils.wire [-] Added IP Routes for network 10.197.0.1/25 _wire_lrp_port_underlay /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/utils/wire.py:621
2024-05-14 02:54:33.716 7 DEBUG ovn_bgp_agent.drivers.openstack.nb_ovn_bgp_driver [-] Adding BGP route for tenant IP(s) ['10.197.0.74'] on chassis w2cosl0011104.heb.com _expose_remote_ip /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/nb_ovn_bgp_driver.py:639
2024-05-14 02:54:33.718 7 DEBUG ovn_bgp_agent.drivers.openstack.nb_ovn_bgp_driver [-] Added BGP route for tenant IP(s) ['10.197.0.74'] on chassis w2cosl0011104.heb.com _expose_remote_ip /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/nb_ovn_bgp_driver.py:645
2024-05-14 02:54:33.718 7 DEBUG ovn_bgp_agent.drivers.openstack.nb_ovn_bgp_driver [-] Adding BGP route for tenant IP(s) ['10.197.0.50'] on chassis w2cosl0011104.heb.com _expose_remote_ip /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/nb_ovn_bgp_driver.py:639
2024-05-14 02:54:33.720 7 DEBUG ovn_bgp_agent.drivers.openstack.nb_ovn_bgp_driver [-] Added BGP route for tenant IP(s) ['10.197.0.50'] on chassis w2cosl0011104.heb.com _expose_remote_ip /var/lib/kolla/venv/lib64/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/nb_ovn_bgp_driver.py:645
2024-05-14 02:54:33.720 7 DEBUG oslo_concurrency.lockutils [-] Lock "nbbgp" "released" by "ovn_bgp_agent.drivers.openstack.nb_ovn_bgp_driver.NBOVNBGPDriver.expose_subnet" :: held 0.041s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:423

The routing table from br-ex.

# ip route show table 53
default dev br-ex scope link
10.196.3.74 dev br-ex scope link
10.197.0.0/25 via 10.196.3.74 dev br-ex

Address list from bgp-nic:

# ip addr show bgp-nic
1864: bgp-nic: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue master bgp-vrf state UNKNOWN group default qlen 1000
    link/ether fe:d5:8e:33:76:b4 brd ff:ff:ff:ff:ff:ff
    inet 10.196.3.74/32 scope global bgp-nic
       valid_lft forever preferred_lft forever
    inet 10.197.0.74/32 scope global bgp-nic
       valid_lft forever preferred_lft forever
    inet 10.197.0.50/32 scope global bgp-nic
       valid_lft forever preferred_lft forever
    inet6 fe80::fcd5:8eff:fe33:76b4/64 scope link
       valid_lft forever preferred_lft forever

# ping 10.197.0.74
PING 10.197.0.74 (10.197.0.74) 56(84) bytes of data.
From 169.254.0.1 icmp_seq=1 Destination Host Unreachable

bgp-agent.conf for ovn-bgp-agent

[DEFAULT]
debug = True
log_file = /var/log/kolla/openvswitch/ovn-bgp-agent.log
ovsdb_connection = unix:/var/run/openvswitch/db.sock
driver = nb_ovn_bgp_driver
exposing_method = underlay
bgp_AS = 64593
bgp_router_id = 99.98.1.3
evpn_nic = br-ex
evpn_local_ip = 99.98.1.3
expose_tenant_networks = True
address_scopes = eb5af6f5-2b92-4c89-87ca-7c7964ca70bd
require_snat_disabled_for_tenant_networks = True

I observe the following routes in the environment:

B>* 10.197.0.50/32 [20/0] is directly connected, bgp-nic (vrf bgp-vrf), weight 1, 00:04:29
B>* 10.197.0.74/32 [20/0] is directly connected, bgp-nic (vrf bgp-vrf), weight 1, 00:04:29

The IPs are not reachable at the moment.

This behavior also occurs if I use subnet as the method. The only difference is that the 10.197.0.50 and 10.197.0.74 are not on the bgp-nic interface.

The route does appear on br-ex (in our case table 53).

10.197.0.0/25 via 10.196.3.74 dev br-ex

The router gateway appears on bgp-nic, but it is not reachable.

    inet 10.196.3.74/32 scope global bgp-nic
       valid_lft forever preferred_lft forever

Even if I add an address for 10.197.0.1/25 or 10.197.0.50/32 onto bgp-nic, I still do not have reachability.

It seems the NB driver for exposing tenant networks is broken.

On the flip side, if we use the ovn_bgp_driver (sb) - We see the VMs exposed and reachable; however, in order to reach additional IP addresses on the VMs (via allowed address pair) we have to add 10.197.0.1/25 to bgp-nic on the network node that has the router.