Ran some additional tests with Przemyslaw, the following can be observed:
1) ssh from octavia to a VM on lb-mgmt-net: the issue IS present
2) ssh between 2 VMs attached to lb-mgmt-net, both located on the same compute node: no issue
3) ssh from a VM to an amphora instance: both attached to lb-mgmt-net and both located on the same compute nodes: no issue
4) ssh from a VM to an amphora instance, both attached to lb-mgmt-net, each VM hosted on different compute nodes: no issue
5) ssh between different octavia units over the tunnel network (via o-hm0 ports): the issue is present
6) iperf3 (TCP or UDP) between different octavia units over the tunnel network (via o-hm0 ports): the issue is present
TCP SYN/SYN-ACK/ACK seems to work but the bandwidth is reported as almost zero.
7) iperf3 between different octavia units over the "overlay" space network: the issue is not present
8) an ICMP test with 100k packet flood (packet size close to standard MTU): the issue is not present (no packets lost).
So far it seems to be happening on the tunnel network only and only when control nodes are in the mix.
We need to narrow this down further, probably with more OVS/OVN level tracing and packet flow simulations.
The original issue looks like the result of impaired transport connectivity since UDP health checks are effected too.
Ran some additional tests with Przemyslaw, the following can be observed:
1) ssh from octavia to a VM on lb-mgmt-net: the issue IS present
2) ssh between 2 VMs attached to lb-mgmt-net, both located on the same compute node: no issue
3) ssh from a VM to an amphora instance: both attached to lb-mgmt-net and both located on the same compute nodes: no issue
4) ssh from a VM to an amphora instance, both attached to lb-mgmt-net, each VM hosted on different compute nodes: no issue
5) ssh between different octavia units over the tunnel network (via o-hm0 ports): the issue is present
6) iperf3 (TCP or UDP) between different octavia units over the tunnel network (via o-hm0 ports): the issue is present
TCP SYN/SYN-ACK/ACK seems to work but the bandwidth is reported as almost zero.
7) iperf3 between different octavia units over the "overlay" space network: the issue is not present
8) an ICMP test with 100k packet flood (packet size close to standard MTU): the issue is not present (no packets lost).
So far it seems to be happening on the tunnel network only and only when control nodes are in the mix.
We need to narrow this down further, probably with more OVS/OVN level tracing and packet flow simulations.
The original issue looks like the result of impaired transport connectivity since UDP health checks are effected too.