Comment 6 for bug 2052485

Eduardo, I may have found something that answers one of my questions to you and may explain the issue

>>- br-ex (dest MAC modified to br-ex MAC due to br-ex flows)
>Can you explain why/how this happens?

One of my colleagues turned me on to this Redhat blog post from 2022 which is clearly heavily influenced or done in conjunction with Luis Tomas's blog posts around the same time. https://developers.redhat.com/articles/2022/09/22/learn-about-new-bgp-capabilities-red-hat-openstack-17#data_plane

The post contains this statement refering to one of the tasks that ovn-bgp-agent is supposed to complete:
"For egress traffic, add flows that change the destination MAC address to that of the provider bridge, so that the kernel will forward the traffic using the default outgoing ECMP routes:"

It then goes on to use this command 'sudo ovs-ofctl dump-flows br-ex' to show that flow rules have indeed been added to rewrite the mac addr for v4 and v6 traffic egressing from the br-int bridge to the br-ex bridge via the provnet patch.

I have repeated this same command on my two hypervisors and I do not have those rules in place, only the statistics summary:
vagrant@rack-1-host-1:~$ sudo ovs-ofctl --protocols=OpenFlow15 dump-flows br-ex
 cookie=0x0, duration=1725926.213s, table=0, n_packets=111160, n_bytes=21586664, idle_age=17, priority=0 actions=NORMAL

vagrant@rack-1-host-2:~$ sudo ovs-ofctl --protocols=OpenFlow15 dump-flows br-ex
 cookie=0x0, duration=1361525.650s, table=0, n_packets=51292, n_bytes=9421634, idle_age=14, priority=0 actions=NORMAL

Now you may notice that my command differs from the one in blog as I've added the --protocols=OpenFlow15 flag. If I don't do this, i get the following error:
vagrant@rack-1-host-1:~$ sudo ovs-ofctl dump-flows br-ex
2024-02-07T20:42:29Z|00001|vconn|WARN|unix:/usr/local/var/run/openvswitch/br-ex.mgmt: version negotiation failed (we support version 0x01, peer supports versions 0x04, 0x06)
ovs-ofctl: br-ex: failed to connect to socket (Protocol error)

Which made me wonder if ovn-bgp-agent is hitting the same error. It is logging a very similar error:

vagrant@rack-1-host-1:~$ sudo journalctl -u devstack@ovn-bgp-agent | grep negotiation
<snipped lots of matches, only showing some of last set>
Feb 07 20:45:11 rack-1-host-1 ovs-ofctl[3426804]: ovs|00001|vconn|WARN|unix:/usr/local/var/run/openvswitch/br-ex.mgmt: version negotiation failed (we support version 0x01, peer supports versions 0x04, 0x06)
Feb 07 20:45:11 rack-1-host-1 ovs-ofctl[3426807]: ovs|00001|vconn|WARN|unix:/usr/local/var/run/openvswitch/br-ex.mgmt: version negotiation failed (we support version 0x01, peer supports versions 0x04, 0x06)
Feb 07 20:45:11 rack-1-host-1 ovs-ofctl[3426809]: ovs|00001|vconn|WARN|unix:/usr/local/var/run/openvswitch/br-ex.mgmt: version negotiation failed (we support version 0x01, peer supports versions 0x04, 0x06)
<snip>
vagrant@rack-1-host-1:~$ date
Wed Feb 7 08:46:14 PM UTC 2024

Every five minutes this message repeats about 10 total times with what I presume are different thread ids.
Is there something in the way that ovn-bgp-agent is using ovs-ofctl (or all ovs-*tools) to fail because of version mismatch?

FWIW from my command line, only specifying --protocols=OpenFlow=13 or OpenFlow=15 work correctly.