Internal IP leak to physical interface from qrouter in DVR mode

[Removed by request]

Setup: Openstack-Ansible cluster(Rocky - 18.1.8) with computes nodes using DVR. OS version Ubuntu 16.04.6 LTS with kernel 4.15.0-34-generic.

Problem: We can see internal IP leaked without NAT on our physical interface. This happens in TCP communication where client stopped abruptly before the server. The leaked packets are always RST packets.

Steps to reproduce:

TCP Client(192.168.100.24, 10.96.48.159)
TCP Server(192.168.100.20, 10.96.48.207)

Server sends RST packets on connection termination.

Step1: Start the server and client.
Setp2: Stop the client(KeyboardInterrupt) while the server is still in the connection.

tcpdump on the bond interface of the compute node in which the tcp client is running

07:50:35.658208 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [S], seq 3764020836, win 64240, options [mss 1460,sackOK,TS val 2823050719 ecr 0,nop,wscale 7], length 0
07:50:35.658539 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [S.], seq 1750463809, ack 3764020837, win 65160, options [mss 1460,sackOK,TS val 2874529221 ecr 2823050719,nop,wscale 7], length 0
07:50:35.658717 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [.], ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 0
07:50:35.658746 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [P.], seq 1:14, ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 13
07:50:35.658949 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 14, win 509, options [nop,nop,TS val 2874529221 ecr 2823050720], length 0
07:50:35.659113 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [P.], seq 14:32, ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 18
07:50:35.659299 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 32, win 509, options [nop,nop,TS val 2874529221 ecr 2823050720], length 0
07:50:40.729542 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [F.], seq 32, ack 1, win 502, options [nop,nop,TS val 2823055790 ecr 2874529221], length 0
07:50:40.773484 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 33, win 509, options [nop,nop,TS val 2874534335 ecr 2823055790], length 0
07:53:35.732815 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [P.], seq 1:21, ack 33, win 509, options [nop,nop,TS val 2874709290 ecr 2823055790], length 20
07:53:35.732878 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [R.], seq 21, ack 33, win 509, options [nop,nop,TS val 2874709291 ecr 2823055790], length 0

07:53:35.733668 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0

tcpdump on the bond interface of the compute node in which the tcp server is running

07:50:35.658302 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [S], seq 3764020836, win 64240, options [mss 1460,sackOK,TS val 2823050719 ecr 0,nop,wscale 7], length 0
07:50:35.658589 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [S.], seq 1750463809, ack 3764020837, win 65160, options [mss 1460,sackOK,TS val 2874529221 ecr 2823050719,nop,wscale 7], length 0
07:50:35.658811 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [.], ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 0
07:50:35.658901 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [P.], seq 1:14, ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 13
07:50:35.658998 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 14, win 509, options [nop,nop,TS val 2874529221 ecr 2823050720], length 0
07:50:35.659205 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [P.], seq 14:32, ack 1, win 502, options [nop,nop,TS val 2823050720 ecr 2874529221], length 18
07:50:35.659350 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 32, win 509, options [nop,nop,TS val 2874529221 ecr 2823050720], length 0
07:50:40.729633 IP 10.96.48.159.36394 > 10.96.48.207.5005: Flags [F.], seq 32, ack 1, win 502, options [nop,nop,TS val 2823055790 ecr 2874529221], length 0
07:50:40.773533 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [.], ack 33, win 509, options [nop,nop,TS val 2874534335 ecr 2823055790], length 0
07:53:35.732868 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [P.], seq 1:21, ack 33, win 509, options [nop,nop,TS val 2874709290 ecr 2823055790], length 20
07:53:35.732898 IP 10.96.48.207.5005 > 10.96.48.159.36394: Flags [R.], seq 21, ack 33, win 509, options [nop,nop,TS val 2874709291 ecr 2823055790], length 0

07:53:35.733767 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.734408 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.734602 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.734748 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.734873 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.734973 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735073 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735171 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735269 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735366 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735464 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735561 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735662 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735776 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735877 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.735975 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.736073 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.736171 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.736269 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.736367 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0
07:53:35.736465 IP 192.168.100.24.36394 > 10.96.48.207.5005: Flags [R], seq 3764020869, win 0, length 0

[Ryan Tidwell (ryan-tidwell)]

I wonder if this is related to some of the address scope / DVR fast-exit work we've done. That work enables tenant subnet traffic to egress without NAT if the address scopes on both the tenant and external network match. I wonder if this is related to some incomplete or buggy implementation of the DVR fast-exit work.

[Brian Haley (brian-haley)]

Ryan - I was thinking the same thing. Had seen this previously on Newton and it had to do with an iptables address scope rule, and it only seemed to be the TCP Reset that was seen externally. If it's easy enough to reproduce someone should be able to track this down.

[Chenjun Shen (cshen)]

Hello,

where could I find the code of DVR fast-exit work?

[Removed by request (removed3304751)]

Hello,

Any progress regarding this issue?? We have this issue regularly in our environments where TCP communication as mentioned in the description happens.