Nova instances on devstack cannot access the external network / the internet

[Goutham Pacha Ravi]

This issue is reproducible in CI as well as locally. A local.conf file that can reproduce this issue is here: https://paste.opendev.org/show/808017/

Devstack is setup with OVN as the Neutron "backend" and default networking settings. Devstack installation completes fine, however, the following workflow no longer works:

1) Create a private network and route it to the external network to provide instances access to the internet

openstack network create private
openstack subnet create --network private --dns-nameserver 10.0.0.1 --gateway 172.20.0.1 --subnet-range 172.20.0.0/16 private-subnet
openstack router create private-router
openstack router set private-router --external-gateway public
openstack router add subnet private-router private-subnet

2) Create a VM on the private network

openstack security group create pingable
openstack security group rule create --proto icmp pingable
openstack security group rule create --proto tcp --dst-port 22 pingable

openstack server create --flavor m1.tiny --image cirros --nic net-id=private --security-group pingable --key-name stack testvm

3) Add a floating IP to the VM

FLOATING_IP=$(openstack floating ip create public -f value -c id)
openstack server add floating ip testvm $FLOATING_IP

4) SSH into the VM via floating IP

   $ ssh cirros@$FLOATING_IP

5) Ping 8.8.8.8 (google DNS), or any external IP address from the VM - fails

   $ ping -w 10 8.8.8.8
   PING 8.8.8.8 (8.8.8.8): 56 data bytes

   --- 8.8.8.8 ping statistics ---
   11 packets transmitted, 0 packets received, 100% packet loss

6) Traceroute shows the packet hanging at the gateway:

   $ traceroute -w 10 8.8.8.8
   traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 46 byte packets
   1 10.1.0.1 (10.1.0.1) 3.689 ms 0.669 ms 0.673 ms
   2 172.24.5.1 (172.24.5.1) 1.089 ms 0.059 ms 0.042 ms
   3 * * *
   4 * * *
   5 * * *
   6 * * *
   7^C

6) Ping the devstack host from the VM - succeeds

  $ $ ping -c 5 192.168.10.197
   PING 192.168.10.197 (192.168.10.197): 56 data bytes
   64 bytes from 192.168.10.197: seq=0 ttl=63 time=0.528 ms
   64 bytes from 192.168.10.197: seq=1 ttl=63 time=1.096 ms
   64 bytes from 192.168.10.197: seq=2 ttl=63 time=0.871 ms
   64 bytes from 192.168.10.197: seq=3 ttl=63 time=1.123 ms
   64 bytes from 192.168.10.197: seq=4 ttl=63 time=0.982 ms

   --- 192.168.10.197 ping statistics ---
   5 packets transmitted, 5 packets received, 0% packet loss
   round-trip min/avg/max = 0.528/0.920/1.123 ms

Some more debug information:
----------------------------

devstack host ip and route info:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:21:24:63 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.197/24 brd 192.168.10.255 scope global dynamic ens3
       valid_lft 82896sec preferred_lft 82896sec
    inet6 fe80::f816:3eff:fe21:2463/64 scope link
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:54:2e:1b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:54:2e:1b brd ff:ff:ff:ff:ff:ff
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 9a:63:5b:ac:0a:4a brd ff:ff:ff:ff:ff:ff
6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether a6:bb:74:e8:24:4a brd ff:ff:ff:ff:ff:ff
    inet 172.24.5.1/24 scope global br-ex
       valid_lft forever preferred_lft forever
    inet6 2001:db8::2/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::a4bb:74ff:fee8:244a/64 scope link
       valid_lft forever preferred_lft forever
7: br-int: <BROADCAST,MULTICAST> mtu 1372 qdisc noop state DOWN group default qlen 1000
    link/ether 12:0d:35:4c:b3:41 brd ff:ff:ff:ff:ff:ff
8: tapf31cd6c2-2c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1372 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000
    link/ether fe:16:3e:2a:3f:32 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe2a:3f32/64 scope link
       valid_lft forever preferred_lft forever
9: tapf6bbd649-60@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 46:57:95:93:fd:27 brd ff:ff:ff:ff:ff:ff link-netns ovnmeta-f6bbd649-6cbc-4cd4-bd95-ac491e2beca3
    inet6 fe80::4457:95ff:fe93:fd27/64 scope link
       valid_lft forever preferred_lft forever
10: tape8fa06a7-8b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1372 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000
    link/ether fe:16:3e:59:4a:6f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe59:4a6f/64 scope link
       valid_lft forever preferred_lft forever

$ ip route show
default via 192.168.10.1 dev ens3 proto dhcp src 192.168.10.197 metric 100
169.254.169.254 via 192.168.10.3 dev ens3 proto dhcp src 192.168.10.197 metric 100
172.24.5.0/24 dev br-ex proto kernel scope link src 172.24.5.1
192.168.10.0/24 dev ens3 proto kernel scope link src 192.168.10.197
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
manila-storage all -- anywhere anywhere
LIBVIRT_INP all -- anywhere anywhere

Chain FORWARD (policy ACCEPT)
target prot opt source destination
LIBVIRT_FWX all -- anywhere anywhere
LIBVIRT_FWI all -- anywhere anywhere
LIBVIRT_FWO all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
LIBVIRT_OUT all -- anywhere anywhere

Chain LIBVIRT_FWI (1 references)
target prot opt source destination
ACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
target prot opt source destination
ACCEPT all -- 192.168.122.0/24 anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere

Chain LIBVIRT_INP (1 references)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:67

Chain LIBVIRT_OUT (1 references)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootpc
ACCEPT tcp -- anywhere anywhere tcp dpt:68

Chain manila-storage (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp dpt:nfs
ACCEPT tcp -- anywhere anywhere tcp dpt:sunrpc
ACCEPT tcp -- anywhere anywhere tcp dpt:32803
ACCEPT tcp -- anywhere anywhere tcp dpt:892
ACCEPT tcp -- anywhere anywhere tcp dpt:875
ACCEPT tcp -- anywhere anywhere tcp dpt:662
ACCEPT udp -- anywhere anywhere udp dpt:sunrpc
ACCEPT udp -- anywhere anywhere udp dpt:32769
ACCEPT udp -- anywhere anywhere udp dpt:892
ACCEPT udp -- anywhere anywhere udp dpt:875
ACCEPT udp -- anywhere anywhere udp dpt:662

[Goutham Pacha Ravi (gouthamr)]

$ sudo ovn-nbctl show
switch 6dca0164-8804-4554-8081-5b87c6b86413 (neutron-093a98a0-f54a-43c4-a890-c07779e5f4f7) (aka public)
    port a5b21609-fcc1-4b64-babc-96902f34e865
        type: router
        router-port: lrp-a5b21609-fcc1-4b64-babc-96902f34e865
    port provnet-6574c3b4-0247-43e4-ab57-9e21d922cb82
        type: localnet
        addresses: ["unknown"]
    port df084bb1-72c4-4cea-aa0b-360f6f67990b
        type: localport
        addresses: ["fa:16:3e:92:d9:20"]
switch 18e825e5-05e9-4547-88d7-14b0453a8298 (neutron-f6bbd649-6cbc-4cd4-bd95-ac491e2beca3) (aka private)
    port 0e416592-421d-42dd-97d3-12dc4bc4077e
        type: router
        router-port: lrp-0e416592-421d-42dd-97d3-12dc4bc4077e
    port 519e17f3-e528-45c6-8b40-7b0a06023e4f
        type: router
        router-port: lrp-519e17f3-e528-45c6-8b40-7b0a06023e4f
    port f31cd6c2-2cc5-4418-a5af-7b1aa738e5d8
        addresses: ["fa:16:3e:2a:3f:32 10.1.0.16 fd45:33a1:2ff6:0:f816:3eff:fe2a:3f32"]
    port e8fa06a7-8bbe-4e45-abab-1a9b402351fc
        addresses: ["fa:16:3e:59:4a:6f 10.1.0.17 fd45:33a1:2ff6:0:f816:3eff:fe59:4a6f"]
    port c60af4eb-b0df-4f50-9f46-2b19fcdbcb09
        type: localport
        addresses: ["fa:16:3e:cf:cc:8a 10.1.0.2 fd45:33a1:2ff6:0:f816:3eff:fecf:cc8a"]
switch 982c617b-cc3d-4f32-b7e8-9a0c6217907a (neutron-3ef1b439-4c78-4735-84b4-816e402d103d) (aka shared)
    port 1c00d97f-9741-4ed0-90cf-3bb6fa077323
        type: localport
        addresses: ["fa:16:3e:f7:76:fc 192.168.233.2"]
router a1f9d9fb-643e-477c-8752-df31ed71afa0 (neutron-eba9a0d1-531b-4ed9-a1ec-b9f2964b3128) (aka router1)
    port lrp-519e17f3-e528-45c6-8b40-7b0a06023e4f
        mac: "fa:16:3e:44:48:a5"
        networks: ["fd45:33a1:2ff6::1/64"]
    port lrp-a5b21609-fcc1-4b64-babc-96902f34e865
        mac: "fa:16:3e:4b:56:7a"
        networks: ["172.24.5.84/24", "2001:db8::1/64"]
        gateway chassis: [376573ca-1613-4238-8959-5f909bc053c5]
    port lrp-0e416592-421d-42dd-97d3-12dc4bc4077e
        mac: "fa:16:3e:51:f3:37"
        networks: ["10.1.0.1/26"]
    nat 31987bed-4780-4891-b718-0f87e850b37d
        external ip: "172.24.5.17"
        logical ip: "10.1.0.16"
        type: "dnat_and_snat"
    nat 62d784d9-4c3c-4f79-b789-2c4b11eaf46c
        external ip: "172.24.5.84"
        logical ip: "10.1.0.0/26"
        type: "snat"
    nat 7346c3e5-a9b0-49c3-afe8-234b0cf163c6
        external ip: "172.24.5.193"
        logical ip: "10.1.0.17"
        type: "dnat_and_snat"

[Goutham Pacha Ravi (gouthamr)]

$ sudo ovn-sbctl show
Chassis "376573ca-1613-4238-8959-5f909bc053c5"
    hostname: zorilla-xena
    Encap geneve
        ip: "192.168.10.197"
        options: {csum="true"}
    Port_Binding "f31cd6c2-2cc5-4418-a5af-7b1aa738e5d8"
    Port_Binding "e8fa06a7-8bbe-4e45-abab-1a9b402351fc"
    Port_Binding cr-lrp-a5b21609-fcc1-4b64-babc-96902f34e865

[Slawek Kaplonski (slaweq)]

I think we found the root cause with Rodolfo. In devstack, when L3 agent is used there is NAT rule created https://github.com/openstack/devstack/blob/e102559f87b3b2afb5ad0aa874f9bfa98c269624/lib/neutron_plugins/services/l3#L112 to provide NATed connectivity to external network.

In case when OVN agent is used, that is not done thus there is no NAT done and no connectivity.

I will propose patch to Devstack to fix that in the lib/neutron_plugins/ovn_agent

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/devstack/+/806267

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/806267
Committed: https://opendev.org/openstack/devstack/commit/b1a89eb80be83fe8c47eeb0431d85a8452e3c70b
Submitter: "Zuul (22348)"
Branch: master

commit b1a89eb80be83fe8c47eeb0431d85a8452e3c70b
Author: Slawek Kaplonski <email address hidden>
Date: Thu Aug 26 21:42:32 2021 +0200

    Configure access to physical network also with ML2/OVN backend

    Neutron L3 module in Devstack has way to conigure access to physical
    network on the node. It can put physical interface to the physical
    bridge or, in case when such physical device isn't set, it creates
    NAT rule in iptables.

    There was missing the same operation for ML2/OVN backend as L3 agent is
    not used there at all.

    This patch adds the same to be done in both L3 agent and ovn_agent
    modules.

    Closes-Bug: #1939627
    Change-Id: I9e558d1d5d3edbce9e7a025ba3c11267f1579820