neutron

Bug #1717302
Comment #8

Comment 8 for bug 1717302

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2017-11-30:

I was able to reproduce this issue locally.

These tests are failing randomly and on further debugging here is what I could see.
In the two node setup.

In Node 1 (Ubuntu-controller) there is one 'VM'
In the Node 2(Ubuntu-compute-new) there are two 'VMs'

Both the VMs in Node2 have floatingIP configured.
Here is the output of the 'router-namespace' iptable rules.

stack@ubuntu-compute-new:~/devstack$ sudo ip netns exec qrouter-6f01678c-64d6-4197-b09d-3285c46207ef bash
root@ubuntu-compute-new:~/devstack# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-POSTROUTING ! -i rfp-6f01678c-6 ! -o rfp-6f01678c-6 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 192.168.100.100/32 -i rfp-6f01678c-6 -j DNAT --to-destination 10.0.0.13
-A neutron-l3-agent-PREROUTING -d 192.168.100.114/32 -i rfp-6f01678c-6 -j DNAT --to-destination 10.0.0.14
-A neutron-l3-agent-float-snat -s 10.0.0.13/32 -j SNAT --to-source 192.168.100.100
-A neutron-l3-agent-float-snat -s 10.0.0.14/32 -j SNAT --to-source 192.168.100.114
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
root@ubuntu-compute-new:~/devstack#

But What I see in the 'Fip namespace' is that the "10.0.0.13" IP is seen within the Fipnamespace responding to a FloatingIP.

stack@ubuntu-compute-new:~$ sudo ip netns exec fip-5c94b420-0b1f-4025-864a-9209d8e7211f tcpdump -i any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
^C19:50:32.073635 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 54785, seq 0, length 64
19:50:35.578246 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 55553, seq 0, length 64
19:50:39.153168 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 56321, seq 0, length 64
19:50:42.790410 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 57089, seq 0, length 64
19:50:46.368505 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 57857, seq 0, length 64
19:50:49.982396 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 58625, seq 0, length 64
19:50:53.553890 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 59393, seq 0, length 64
19:50:57.005240 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 60161, seq 0, length 64
19:51:00.557693 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 60929, seq 0, length 64
19:51:04.045430 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 61697, seq 0, length 64
19:51:07.579294 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 62465, seq 0, length 64
19:51:11.229360 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 63233, seq 0, length 64
19:51:14.758319 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 64001, seq 0, length 64
19:51:18.572113 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 64769, seq 0, length 64
19:51:34.087878 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 2, seq 0, length 64
19:51:50.009333 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 770, seq 0, length 64
19:53:11.005206 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 1538, seq 0, length 64

17 packets captured
17 packets received by filter
0 packets dropped by kernel

So from the FIP namespace it is forwarded to the 192.168.100.109 which is the gateway port IP.

But the Gateway port configured in "Node1" is not receiving the packets.

If you see the packets inside the "SNAT namespace" this is what I see.
stack@ubuntu-controller:~/devstack$ sudo ip netns exec snat-6f01678c-64d6-4197-b09d-3285c46207ef tcpdump -i any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
^C19:48:41.764206 IP 10.0.0.8 > 192.168.100.100: ICMP echo request, id 30721, seq 0, length 64
19:48:41.764222 IP 192.168.100.109 > 10.0.0.13: ICMP echo request, id 30721, seq 0, length 64
19:48:45.337621 IP 10.0.0.8 > 192.168.100.100: ICMP echo request, id 31489, seq 0, length 64
19:48:45.337649 IP 192.168.100.109 > 10.0.0.13: ICMP echo request, id 31489, seq 0, length 64
19:48:48.972499 IP 10.0.0.8 > 192.168.100.100: ICMP echo request, id 32257, seq 0, length 64
19:48:48.972519 IP 192.168.100.109 > 10.0.0.13: ICMP echo request, id 32257, seq 0, length 64
19:48:52.540429 IP 10.0.0.8 > 192.168.100.100: ICMP echo request, id 33025, seq 0, length 64
19:48:52.540451 IP 192.168.100.109 > 10.0.0.13: ICMP echo request, id 33025, seq 0, length 64
19:48:56.036427 IP 10.0.0.8 > 192.168.100.100: ICMP echo request, id 33793, seq 0, length 64
19:48:56.036445 IP 192.168.100.109 > 10.0.0.13: ICMP echo request, id 33793, seq 0, length 64
19:48:59.675894 IP 10.0.0.8 > 192.168.100.100: ICMP echo request, id 34561, seq 0, length 64
19:48:59.675916 IP 192.168.100.109 > 10.0.0.13: ICMP echo request, id 34561, seq 0, length 64
19:49:03.175191 IP 10.0.0.8 > 192.168.100.100: ICMP echo request, id 35329, seq 0, length 64
19:49:03.175211 IP 192.168.100.109 > 10.0.0.13: ICMP echo request, id 35329, seq 0, length 64
19:49:06.758560 IP 10.0.0.8 > 192.168.100.100: ICMP echo request, id 36097, seq 0, length 64
19:49:06.758600 IP 192.168.100.109 > 10.0.0.13: ICMP echo request, id 36097, seq 0, length 64
19:49:10.483643 IP 10.0.0.8 > 192.168.100.100: ICMP echo request, id 36865, seq 0, length 64
19:49:10.483662 IP 192.168.100.109 > 10.0.0.13: ICMP echo request, id 36865, seq 0, length 64
19:49:14.081248 IP 10.0.0.8 > 192.168.100.100: ICMP echo request, id 37889, seq 0, length 64
19:49:14.081264 IP 192.168.100.109 > 10.0.0.13: ICMP echo request, id 37889, seq 0, length 64
19:49:17.493013 IP 10.0.0.8 > 192.168.100.100: ICMP echo request, id 38657, seq 0, length 64
19:49:17.493028 IP 192.168.100.109 > 10.0.0.13: ICMP echo request, id 38657, seq 0, length 64

22 packets captured
22 packets received by filter
0 packets dropped by kernel
stack@ubuntu-controller:~/devstack$

stack@ubuntu-controller:~/devstack$ sudo ip netns exec snat-6f01678c-64d6-4197-b09d-3285c46207ef bash
root@ubuntu-controller:~/devstack# ifconfig
lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

qg-774bcb71-fa Link encap:Ethernet HWaddr fa:16:3e:70:c5:48
          inet addr:192.168.100.109 Bcast:192.168.100.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe70:c548/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:66 errors:0 dropped:0 overruns:0 frame:0
          TX packets:70 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:4148 (4.1 KB) TX bytes:4597 (4.5 KB)

sg-ea74de23-53 Link encap:Ethernet HWaddr fa:16:3e:a2:ce:37
          inet addr:10.0.0.6 Bcast:10.0.0.15 Mask:255.255.255.240
          inet6 addr: fe80::f816:3eff:fea2:ce37/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
          RX packets:190 errors:0 dropped:0 overruns:0 frame:0
          TX packets:101 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:16644 (16.6 KB) TX bytes:9292 (9.2 KB)

root@ubuntu-controller:~/devstack#

There is no floatingIP configured in the 'Node1'(ubuntu-controller) node.

stack@ubuntu-controller:~$ sudo ip netns exec qrouter-6f01678c-64d6-4197-b09d-3285c46207ef bash
root@ubuntu-controller:~# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-POSTROUTING ! -i rfp-6f01678c-6 ! -o rfp-6f01678c-6 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
root@ubuntu-controller:~# tcpdump -i any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
root@ubuntu-controller:~#

I see the reply and response not getting in sync.

I was able to reproduce this issue locally.

These tests are failing randomly and on further debugging here is what I could see.
In the two node setup.

In Node 1 (Ubuntu-controller) there is one 'VM'
In the Node 2(Ubuntu-compute-new) there are two 'VMs'

Both the VMs in Node2 have floatingIP configured. 
Here is the output of the 'router-namespace' iptable rules.

But What I see in the 'Fip namespace' is that the "10.0.0.13" IP is seen within the Fipnamespace responding to a FloatingIP.

stack@ubuntu-compute-new:~$ sudo ip netns exec fip-5c94b420-0b1f-4025-864a-9209d8e7211f tcpdump -i any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
 ^C19:50:32.073635 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 54785, seq 0, length 64
19:50:35.578246 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 55553, seq 0, length 64
19:50:39.153168 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 56321, seq 0, length 64
19:50:42.790410 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 57089, seq 0, length 64
19:50:46.368505 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 57857, seq 0, length 64
19:50:49.982396 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 58625, seq 0, length 64
19:50:53.553890 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 59393, seq 0, length 64
19:50:57.005240 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 60161, seq 0, length 64
19:51:00.557693 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 60929, seq 0, length 64
19:51:04.045430 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 61697, seq 0, length 64
19:51:07.579294 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 62465, seq 0, length 64
19:51:11.229360 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 63233, seq 0, length 64
19:51:14.758319 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 64001, seq 0, length 64
19:51:18.572113 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 64769, seq 0, length 64
19:51:34.087878 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 2, seq 0, length 64
19:51:50.009333 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 770, seq 0, length 64
19:53:11.005206 IP 10.0.0.13 > 192.168.100.109: ICMP echo reply, id 1538, seq 0, length 64

17 packets captured
17 packets received by filter
0 packets dropped by kernel

So from the FIP namespace it is forwarded to the 192.168.100.109 which is the gateway port IP.

But the Gateway port configured in "Node1" is not receiving the packets.

22 packets captured
22 packets received by filter
0 packets dropped by kernel
stack@ubuntu-controller:~/devstack$

stack@ubuntu-controller:~/devstack$ sudo ip netns exec snat-6f01678c-64d6-4197-b09d-3285c46207ef bash
root@ubuntu-controller:~/devstack# ifconfig
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

qg-774bcb71-fa Link encap:Ethernet  HWaddr fa:16:3e:70:c5:48  
          inet addr:192.168.100.109  Bcast:192.168.100.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe70:c548/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:66 errors:0 dropped:0 overruns:0 frame:0
          TX packets:70 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:4148 (4.1 KB)  TX bytes:4597 (4.5 KB)

sg-ea74de23-53 Link encap:Ethernet  HWaddr fa:16:3e:a2:ce:37  
          inet addr:10.0.0.6  Bcast:10.0.0.15  Mask:255.255.255.240
          inet6 addr: fe80::f816:3eff:fea2:ce37/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:190 errors:0 dropped:0 overruns:0 frame:0
          TX packets:101 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:16644 (16.6 KB)  TX bytes:9292 (9.2 KB)

root@ubuntu-controller:~/devstack#

There is no floatingIP configured in the 'Node1'(ubuntu-controller) node.

I see the reply and response not getting in sync.