you can end up in a state where qvo* interfaces aren't owned by ovs which results in a dangling connection

[Ian Kumlien]

While upgrading to rocky, we ended up with a broken openvswitch infrastructure and moved back to the old openvswitch.

We ended up with new machines working, old machines didn't and it took a while to realize that we had qvo* interfaces that not only wasn't plugged but also wasn't owned by ovs-system - basically the virtual equivalent of forgetting to plug in the cable ;)

This was quickly addressed by running this bash-ism on all nodes:
for x in `ip a |grep qvo |grep @qvb |grep -v ovs-system | awk '{ print $2 '}` ; do y=${x%%"@"*} && ip link delete $y ; done ; docker restart nova_compute

However, nova could pretty easily sanity check this =)

[sean mooney (sean-k-mooney)]

this might be somthing that could be added to the exsiting
neutron-ovs-cleanup script
that is generated by this entry point https://github.com/openstack/neutron/blob/master/setup.cfg#L49
and impmeneted here https://github.com/openstack/neutron/blob/master/neutron/cmd/ovs_cleanup.py

but this should not live in nova.

[Ian Kumlien (pomac)]

Is this script run on each startup? The issue is that nothing will work until it's been run.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Ian:

Can you provide which was the original version?

Did you move from iptables hybrid driver to OVS firewall? Or did you just remove the hybrid driver? In this case, there is a section describing how to handle this upgrade: https://github.com/openstack/neutron/blob/master/doc/source/contributor/internals/openvswitch_firewall.rst#upgrade-path-from-iptables-hybrid-driver

Regards.

[Ian Kumlien (pomac)]

No, we upgraded to a version of openvswitch that didn't work, it kept busylooping and didn't actually handle any packets - so we reverted, but connections was left in this disconnected state.

We have had this happening several times since then as well - IMHO nova should check the interfaces to see that they are *connected* on startup.

[Ian Kumlien (pomac)]

(note, we use the fact that nova will set them up if they are missing - but it will not do a small sanity check)