net/sched: act_ct: Fix promotion of offloaded unreplied tuple

[Tony Duan]

* Explain the bug(s)

Currently UNREPLIED and UNASSURED connections are added to the nf flow table.
This causes the following connection packets to be processed by the flow table
which then skips conntrack_in(), and thus such the connections will remain
UNREPLIED and UNASSURED even if reply traffic is then seen. Even still, the
unoffloaded reply packets are the ones triggering hardware update from new to
established state, and if there aren't any to triger an update and/or previous
update was missed, hardware can get out of sync with sw and still mark packets
as new.

* brief explanation of fixes

  1) Not skipping conntrack_in() for UNASSURED packets, but still
     refresh for hardware, as before the cited patch.
  2) Try and force a refresh by reply-direction packets that update
     the hardware rules from new to established state.
  3) Remove any bidirectional flows that didn't failed to update in
     hardware for re-insertion as bidirectional once any new packet
     arrives.

* How to test

Create OVS bridge with 2 devices mlx5 rep devices.
Enable HW offload and configure regular connection tracking OpenFlow rules:

e.g:
    ovs-ofctl del-flows br-ovs
    ovs-ofctl add-flow br-ovs arp,actions=normal
    ovs-ofctl add-flow br-ovs "table=0, ip,ct_state=-trk actions=ct(table=1)"
    ovs-ofctl add-flow br-ovs "table=1, ip,ct_state=+trk+new actions=ct(commit),normal"
    ovs-ofctl add-flow br-ovs "table=1, ip,ct_state=+trk+est, actions=normal"

Run a UDP single sided connection, e.g iperf without a server:
on mlx5 VF2 iperf -t 1 -c <ip>-u -b 10pps

Optional: In different terminal, while traffic is running, check for offload:
tcpdump -nnepi <RELEVANT_MLX5_REP> udp
and see no iperf udp packets.

Dump conntrack with relevant ip:
cat /proc/net/nf_conntrack | grep -i <ip> | grep "\[UNREPLIED\]" | grep -q "\[OFFLOAD\]"

If tuple appears, tuple was offloaded, yet unreplied.

* What it could break.

The connections remain UNREPLIED and UNASSURED even if reply traffic is seen

[Bartlomiej Zolnierkiewicz (bzolnier)]

This bug is awaiting verification that the linux-bluefield/5.15.0-1019.21 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!