Metadata broken for SR-IOV external ports

[Trent Lloyd]

OpenStack Usurri/OVN SR-IOV instances are unable to connect to the metadata service despite DHCP and normal traffic work.

The 169.254.169.254 metadata route is directed at the DHCP port IP, and no arp reply is received by the VM for this IP. Diagnosis finds that the ARP reply returns from the ovnmeta namespace on the chassis hosting the external port but is dropped inside OVS.

20.03.2-0ubuntu0.20.04.2 backported the following patch:
Do not forward traffic from localport to localnet ports (LP: #1943266)
(d/p/lp-1943266-physical-do-not-forward-traffic-from-localport-to-a-.patch)

This patch broke metadata for SR-IOV external ports and was fixed in 1148580290d0ace803f20aeaa0241dd51c100630 "Don't suppress localport traffic directed to external port":
https://github.com/ovn-org/ovn/commit/1148580290d0ace803f20aeaa0241dd51c100630

[Trent Lloyd (lathiat)]

"Don't suppress localport traffic directed to external port" (https://github.com/ovn-org/ovn/commit/1148580290d0ace803f20aeaa0241dd51c100630) does not appear super trivially back-portable due to depending on among other things logic from https://github.com/ovn-org/ovn/commit/6c8b9a13253248318c666159e486aab0e6c477ff.

For backporting purposes we also need to backport the 21.06 branch’s cherry-pick which has extra logic for flow cleanup/deletion that wasn't needed in the master patch - that commit is https://github.com/ovn-org/ovn/commit/0e70fe417ea0413bef8a1b2803707e27e231f12a

I *think* it could back ported but would need understanding more thoroughly and partly rewriting.