Comment 26 for bug 1817936

from this log , The root cause is found.
probe was in TIME_WAIT connection(10.10.10.3:50538->172.16.192.101:9292)
client send a new connection 10.10.10.3:50538->10.109.43.235:9292, after NAT the output SYN package will be
10.10.10.3:24479->172.16.192.101:9292. the SYN ACK 172.16.192.101:9292->10.10.10.3:24479 will be modified by Mangle to 172.16.192.101:9292->10.10.10.3:50538, but this socket was still in TIME_WAIT state, will be sent INVALID and then drop.

==========================================================================================
describe controller service endpoint STATE
                (10.10.10.3) (10.109.43.235) (172.16.192.101)
     Probe *(50538)------------------------->*(9292) TIME_WAIT
     Client New *(50538)----> *(9292)
     After NAT
     Client *(24479)------------------------->*(9292) SYN_SENT
     IN PREROUTING *(24479)<-------------------------*(9292) SYN_RECV
     INPUT Chain *(50538)<-------------------------*(9292) SYN_RECV

==========================================================================================
tcpdump_port_9292.cap:
No.24288 10.10.10.3:24479->172.16.192.101:9292
No.24289 172.16.192.101:9292->10.10.10.3:24479

conntrack_tcp_port9292:

[1565008856.898709] [UPDATE] tcp 6 120 TIME_WAIT src=10.10.10.3 dst=172.16.192.101 sport=50538 dport=9292 src=172.16.192.101 dst=10.10.10.3 sport=9292 dport=50538 [ASSURED]
[1565008911.774492] [NEW] tcp 6 120 SYN_SENT src=10.10.10.3 dst=10.109.43.235 sport=50538 dport=9292 [UNREPLIED] src=172.16.192.101 dst=10.10.10.3 sport=9292 dport=24479
[1565008911.774522] [UPDATE] tcp 6 60 SYN_RECV src=10.10.10.3 dst=10.109.43.235 sport=50538 dport=9292 src=172.16.192.101 dst=10.10.10.3 sport=9292 dport=24479

because port 50538 is still in TIME_WAIT to release.
New connection NAT can not use this CT. so NAT change to 10.10.10.3:24479->172.16.192.101:9292
10.10.10.3:50538->10.109.43.235:9292 nat to 10.10.10.3:24479->172.16.192.101:9292
kern_9292.log:
mangle change the CT 172.16.192.101:9292->10.10.10.3:24479 to 172.16.192.101:9292->10.10.10.3:50538
because this 172.16.192.101:9292->10.10.10.3:50538 CT is still waitting for DESTORY,
so 172.16.192.101:9292->10.10.10.3:50538 is set invalid. then package is droped

2019-08-05T12:41:51.775 controller-0 kernel: warning [41907.116655] TRACE: mangle:cali-PREROUTING:rule:1 IN=calif380bc70786 OUT= MAC=ee:ee:ee:ee:ee:ee:c6:6d:0e:f5:a5:88:08:00 SRC=172.16.192.101 DST=10.10.10.3 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=9292 DPT=24479 SEQ=2742336307 ACK=2917708675 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A027ADF54027ADF5401030307) MARK=0x40000
2019-08-05T12:41:51.775 controller-0 kernel: warning [41907.116661] TRACE: mangle:INPUT:policy:1 IN=calif380bc70786 OUT= MAC=ee:ee:ee:ee:ee:ee:c6:6d:0e:f5:a5:88:08:00 SRC=172.16.192.101 DST=10.10.10.3 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=9292 DPT=50538 SEQ=2742336307 ACK=2917708675 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A027ADF54027ADF5401030307) MARK=0x40000