FIPs not working with DVR and VLAN provider networks

[Krzysiek Sz]

In latest Tripleo/Ussuri installations with OVN and DVR, instances using floating IPs cannot reach external IPs (outside the floating IP network VLAN).
The case is valid only for floating IPs in DVR scenario, when provider network is of type VLAN. Such instances can communicate only within the provider network VLAN. FIPs work only when DVR is disabled or when provider network is of type flat. Routers and directly attached instances are always pingable from external networks.

Sniffing packets on the provider network reveals that there are ARPs (like "Request who-has 8.8.8.8 tell 10.0.0.59", where 10.0.0.59 is a floating IP), which obviously fail and packets are dropped. This would also explain why the communication works fine when it doesn't need to cross VLANs. It seems that an OVN router sets next hop as the destination IP instead of the gateway.

The comment https://bugzilla.redhat.com/show_bug.cgi?id=1834433#c4 by points to this change as the source of the problem https://github.com/ovn-org/ovn/commit/c0bf32d72f8b893bbe3cb64912b0fd259d71555f

OVN version present in current containers: ovn-20.03.0-4.el8.x86_64