ovs flows aren't cleaned up after switch to iptables firewall under high-load

[Inessa Vasilevskaya]

Seen on: newton devstack, ubuntu 16.04, firewall_driver=openvswitch.

To emulate high load I cleared all quotas, created a security-group A with ~4200 security group rules with remote_group_id pointing to security-group B and booted 2 vms (one with secgroup A and another with secgroup B). Due to https://bugs.launchpad.net/neutron/+bug/1628819 every next VM boot resulted in plenty of ovs flows, so after booting 15 vms and reaching ~23000 flows every other VM would go into ERROR with nova blaming neutron for not providing network for an instance (nova compute logs - http://paste.openstack.org/show/597972/). The ovs-vswitchd logs complained of excessive load as well so my initial guess was that high load was the matter.

After the environment was "heavy loaded" the switch to iptables firewall (and subsequent ovs-agent restart) didn't clean up the generated flows (23407 flows remained), although ovs-agent logs showed that the driver was changed http://paste.openstack.org/show/597978/

[Jakub Libosvar (libosvar)]

Also note that if you switch from openvswitch firewall driver to iptables firewall driver, then security groups won't work as existing instances won't have qbr but tap is plugged directly to br-int. Maybe we should document that migrations of firewalls are not supported yet.

[Inessa Vasilevskaya (ivasilevskaya)]

@jlibosva, yeah, I'm aware of absence of auto migration yet. I just presumed that ovs flows should be cleared anyway on firewall switch. I'm a bit of a loss tere, as I can't find a way to get a working environment (apart from manually destroying all the vms) when ovs-agent is overloaded.