The probe connection action before going to time_wait state. 
Probe connection
controller                                        service                     endpoint                            Chain                      TCP FLAG                           SEQ          ACK
10.10.10.3:50538  ------------------------------->    172.16.192.101:9292     raw:OUTPUT:policy:4               SYN                       2707980036        0
10.10.10.3:50538  <-------------------------------    172.16.192.101:9292    raw:PREROUTING:policy:4     SYN ACK                 1599414185    2707980037
10.10.10.3:50538  ------------------------------->    172.16.192.101:9292    filter:OUTPUT:rule:1                 ACK                       2707980037    1599414186
10.10.10.3:50538  ------------------------------->    172.16.192.101:9292     raw:OUTPUT:policy:4             FIN ACK                 2707980037    1599414186
10.10.10.3:50538  <-------------------------------    172.16.192.101:9292    raw:PREROUTING:policy:4       ACK                      1599414186    2707980038
10.10.10.3:50538  <-------------------------------    172.16.192.101:9292    raw:PREROUTING:policy:4      FIN ACK                1599414186    2707980038
10.10.10.3:50538  ------------------------------->    172.16.192.101:9292     raw:OUTPUT:policy:4               ACK                      2707980038    1599414187

And for the curl command connection with same port 50538: it will be like 
controller                                   service                     endpoint                                 Chain                            TCP FLAG            SEQ          ACK
10.10.10.3:50538  -->  10.109.43.235:9292                                               raw:OUTPUT:policy:4                  SYN           2917708674        0
10.10.10.3:50538  ------------------------------->    172.16.192.101:9292    filter:OUTPUT:rule:1                    SYN           2917708674        0
10.10.10.3:24479  <-------------------------------    172.16.192.101:9292    raw:PREROUTING:policy:4      SYN ACK        2742336307    2917708675
10.10.10.3:50538  <-------------------------------    172.16.192.101:9292    mangle:INPUT:policy:1            SYN ACK        2742336307    2917708675
10.10.10.3:50538  -->  10.109.43.235:9292                                               raw:OUTPUT:policy:4                   ACK           2707980038    1599414187
10.10.10.3:50538  -->  10.109.43.235:9292                                               filter:OUTPUT:rule:1                    ACK           2707980038    1599414187                                          
10.10.10.3:50538  -->  10.109.43.235:9292                                              filter:cali-th-ens6:rule:2         ACK(DROP)     2707980038    1599414187

The last ACK(10.10.10.3:50538  -->  10.109.43.235:9292) SEQ and ACK is same as Probe TIME_WAIT latest ACK’s .  
from https://github.com/torvalds/linux/blob/v3.10/net/ipv4/tcp_ipv4.c#L2002 , it only check (des ip , des port, src ip, and src port)
Because this is not a correct SEQ/ACK , then it is set invalid and then dropped. 

If enabling tcp_tw_recycle, the previous socket should be already closed , then the issue should be gone.