floatingip can not access after associate to instance

[Zhang Jian]

I have deploy openstack neutron component user kolla ansible with the rocky release successfully.
and I enabled the SDN ML2 plugin in neutron ml2_conf.ini。
when I create a baremetal port from a vlan internal network, it the SDN controller can modify the vlan automatically.
And network works normally,shown below:
root@ubuntu:~# ip netns exec qrouter-50c1c5ac-1676-4a9d-ab04-a1333381a700 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
53: qr-66ff06af-8a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:da:e3:3c brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.254/24 brd 192.168.1.255 scope global qr-66ff06af-8a
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feda:e33c/64 scope link
       valid_lft forever preferred_lft forever
54: qg-091949c0-13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:7c:5d:3f brd ff:ff:ff:ff:ff:ff
    inet 36.250.72.178/24 brd 36.250.72.255 scope global qg-091949c0-13
       valid_lft forever preferred_lft forever
    inet 36.250.72.179/32 brd 36.250.72.179 scope global qg-091949c0-13
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe7c:5d3f/64 scope link
       valid_lft forever preferred_lft forever

And I can ping public gateway and internal port from namespace:
root@ubuntu:~# ip netns exec qrouter-50c1c5ac-1676-4a9d-ab04-a1333381a700 ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.594 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=0.178 ms
^C
--- 192.168.1.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1017ms
rtt min/avg/max/mdev = 0.178/0.386/0.594/0.208 ms
root@ubuntu:~# ip netns exec qrouter-50c1c5ac-1676-4a9d-ab04-a1333381a700 ping 36.250.72.177
PING 79.61.92.177 (36.250.72.177) 56(84) bytes of data.
64 bytes from 36.250.72.177: icmp_seq=1 ttl=255 time=0.277 ms
64 bytes from 36.250.72.177: icmp_seq=2 ttl=255 time=0.275 ms
64 bytes from 36.250.72.177: icmp_seq=3 ttl=255 time=0.309 ms
^C
--- 36.250.72.177 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2042ms
rtt min/avg/max/mdev = 0.275/0.287/0.309/0.015 ms

And the instance can alos access exteral network normally
root@instance:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=255 time=0.277 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=255 time=0.275 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=255 time=0.309 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2042ms
rtt min/avg/max/mdev = 0.275/0.287/0.309/0.015 ms

but after I associate a floatingip to this port(floatingip is:36.250.72.180):
neutron floatingip-associate f10a8e0a-3e86-407e-a654-7187ebc16e72 386dc61a-c01c-46ff-b001-eb799b3b6042

I can not access from 36.250.72.180 to instance and instance also can not access to external network.
but from the namespace the network still shown normally.

I doubt the error occurs in my wrong neutron configurations. but in some case, the floatingip can access normally and no longer appears.

I only can reproduce when a create a new neutron router and reassociate the floatingip to a port that attach to this router in a new namespace.

the following show the iptables and ovs configuration when error occurs:

root@ubuntu:~# ip netns exec qrouter-0ccc1435-636d-41b9-912c-2a96c68e6a09 iptables-save
# Generated by iptables-save v1.6.1 on Fri Nov 15 05:16:28 2019
*raw
:PREROUTING ACCEPT [113408:41184050]
:OUTPUT ACCEPT [9442:553311]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
COMMIT
# Completed on Fri Nov 15 05:16:28 2019
# Generated by iptables-save v1.6.1 on Fri Nov 15 05:16:28 2019
*nat
:PREROUTING ACCEPT [2515:147604]
:INPUT ACCEPT [1126:64144]
:OUTPUT ACCEPT [1:84]
:POSTROUTING ACCEPT [1148:66130]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 36.250.72.179/32 -j DNAT --to-destination 192.168.1.8
-A neutron-l3-agent-POSTROUTING ! -i qg-091949c0-13 ! -o qg-091949c0-13 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 36.250.72.178/32 -p tcp -m tcp --dport 36148 -j DNAT --to-destination 192.168.1.8:22
-A neutron-l3-agent-PREROUTING -d 36.250.72.179/32 -j DNAT --to-destination 192.168.1.8
-A neutron-l3-agent-float-snat -s 192.168.1.8/32 -j SNAT --to-source 36.250.72.179
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -o qg-091949c0-13 -j SNAT --to-source 36.250.72.178
-A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 36.250.72.178
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
COMMIT
# Completed on Fri Nov 15 05:16:28 2019
# Generated by iptables-save v1.6.1 on Fri Nov 15 05:16:28 2019
*mangle
:PREROUTING ACCEPT [113408:41184050]
:INPUT ACCEPT [15649:855387]
:FORWARD ACCEPT [97758:40328619]
:OUTPUT ACCEPT [9442:553311]
:POSTROUTING ACCEPT [107200:40881930]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-floatingip - [0:0]
:neutron-l3-agent-mark - [0:0]
:neutron-l3-agent-scope - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A neutron-l3-agent-POSTROUTING -o qg-091949c0-13 -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-mark
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-scope
-A neutron-l3-agent-PREROUTING -m connmark ! --mark 0x0/0xffff0000 -j CONNMARK --restore-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-floatingip
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff
-A neutron-l3-agent-float-snat -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-A neutron-l3-agent-mark -i qg-091949c0-13 -j MARK --set-xmark 0x2/0xffff
-A neutron-l3-agent-scope -i qr-66ff06af-8a -j MARK --set-xmark 0x4000000/0xffff0000
-A neutron-l3-agent-scope -i qg-091949c0-13 -j MARK --set-xmark 0x4000000/0xffff0000
COMMIT
# Completed on Fri Nov 15 05:16:28 2019
# Generated by iptables-save v1.6.1 on Fri Nov 15 05:16:28 2019
*filter
:INPUT ACCEPT [2645:125707]
:FORWARD ACCEPT [8778:536139]
:OUTPUT ACCEPT [1143:66815]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
:neutron-l3-agent-scope - [0:0]
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A neutron-filter-top -j neutron-l3-agent-local
-A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope
-A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
-A neutron-l3-agent-scope -o qr-66ff06af-8a -m mark ! --mark 0x4000000/0xffff0000 -j DROP
COMMIT
# Completed on Fri Nov 15 05:16:28 2019

(openvswitch-db)[root@ubuntu /]# ovs-vsctl show
b60c8aa9-825d-4ede-9d55-2e24f6439a0d
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-tenant
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "eno2"
            Interface "eno2"
        Port br-tenant
            Interface br-tenant
                type: internal
        Port phy-br-tenant
            Interface phy-br-tenant
                type: patch
                options: {peer=int-br-tenant}
    Bridge br-public
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port phy-br-public
            Interface phy-br-public
                type: patch
                options: {peer=int-br-public}
        Port br-public
            Interface br-public
                type: internal
        Port "eno1"
            Interface "eno1"
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "tap70437459-63"
            tag: 15
            Interface "tap70437459-63"
                type: internal
        Port "qr-66ff06af-8a"
            tag: 12
            Interface "qr-66ff06af-8a"
                type: internal
        Port "qr-edf5e7b0-1f"
            tag: 15
            Interface "qr-edf5e7b0-1f"
                type: internal
        Port "qg-320d72e5-13"
            tag: 13
            Interface "qg-320d72e5-13"
                type: internal
        Port int-br-public
            Interface int-br-public
                type: patch
                options: {peer=phy-br-public}
        Port "qg-091949c0-13"
            tag: 13
            Interface "qg-091949c0-13"
                type: internal
        Port int-br-tenant
            Interface int-br-tenant
                type: patch
                options: {peer=phy-br-tenant}
        Port "tap8fb11d88-5a"
            tag: 12
            Interface "tap8fb11d88-5a"
                type: internal
        Port "tapbd651bdf-e5"
            tag: 1
            Interface "tapbd651bdf-e5"
                type: internal
        Port "qr-5eee18b8-35"
            tag: 1
            Interface "qr-5eee18b8-35"
                type: internal

[Lajos Katona (lajos-katona)]

Hi, thanks for reporting. Which SDN controller do you use?

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Did you try disassociating the current floatingip and then reassociating with the new Port that you created from a subnet connected to the new router.
Just to make sure if this is a cleanup issue.

Also can you check if the router-namespace has the correct DNAT rules for the floatingip and the Private IP of the VM ( instance ) port.

Check if you ping reaches the VM by sniffing the external gateway port on the router namespace.
See if it hits the DNAt rule.

[Slawek Kaplonski (slaweq)]

I'm marking this bug as incomplete for now. Please switch it back to "new" when You will provide info which Lajos ans Swami were asking in previous comments.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]