ml2 ovs does not flush iptables switching to FW ovs

[bjolo]

hi,

When switching fw engine from itables to openvswitch and restart the agent, the old iptables rules are not flushed. One has to clean that up by hand or reboot. This is not documented anywhere afaik and it gives very tricky issues that are hard to detect.

#### OVS with FW = openvswithc
# iptables -L | grep neutron
< returns nothing >

#### switching to FW = iptables and restart agent
# iptables -S | grep neutron
-N neutron-filter-top
-N neutron-openvswi-FORWARD
-N neutron-openvswi-INPUT
-N neutron-openvswi-OUTPUT
-N neutron-openvswi-local
-N neutron-openvswi-sg-chain
-N neutron-openvswi-sg-fallback
-A INPUT -j neutron-openvswi-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-openvswi-OUTPUT
-A neutron-filter-top -j neutron-openvswi-local
-A neutron-openvswi-FORWARD -m physdev --physdev-out tapc02b9364-d2 --physdev-is-bridged -m comment --comment "Accept all packets when port security is disabled." -j ACCEPT
-A neutron-openvswi-FORWARD -m physdev --physdev-in tapc02b9364-d2 --physdev-is-bridged -m comment --comment "Accept all packets when port security is disabled." -j ACCEPT
-A neutron-openvswi-INPUT -m physdev --physdev-in tapc02b9364-d2 --physdev-is-bridged -m comment --comment "Accept all packets when port security is disabled." -j ACCEPT
-A neutron-openvswi-sg-chain -j ACCEPT
-A neutron-openvswi-sg-fallback -m comment --comment "Default drop rule for unmatched traffic." -j DROP

#### swtiching back to FW = ovs and restarting the agent, the iptables rules are still there
# iptables -S | grep neutron
-N neutron-filter-top
-N neutron-openvswi-FORWARD
-N neutron-openvswi-INPUT
-N neutron-openvswi-OUTPUT
-N neutron-openvswi-local
-N neutron-openvswi-sg-chain
-N neutron-openvswi-sg-fallback
-A INPUT -j neutron-openvswi-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-openvswi-OUTPUT
-A neutron-filter-top -j neutron-openvswi-local
-A neutron-openvswi-FORWARD -m physdev --physdev-out tapc02b9364-d2 --physdev-is-bridged -m comment --comment "Accept all packets when port security is disabled." -j ACCEPT
-A neutron-openvswi-FORWARD -m physdev --physdev-in tapc02b9364-d2 --physdev-is-bridged -m comment --comment "Accept all packets when port security is disabled." -j ACCEPT
-A neutron-openvswi-INPUT -m physdev --physdev-in tapc02b9364-d2 --physdev-is-bridged -m comment --comment "Accept all packets when port security is disabled." -j ACCEPT
-A neutron-openvswi-sg-chain -j ACCEPT
-A neutron-openvswi-sg-fallback -m comment --comment "Default drop rule for unmatched traffic." -j DROP

########### Expected behavior #############
The agent should check what FW engine is used and check if there is something to clean up
i.e.
if config fw = ovs, check and clean up iptables
if config fw = iptabls, check and clean up ovs fw flows

[Slawek Kaplonski (slaweq)]

I don't think that we should add such cleaning of something which could be made by different driver to the code. Maybe we should simply document that and make sure that operators are aware of such need of cleaning rules during the migration?

[bjolo (bjorn-lofdahl)]

That is one way of doing it of course, and provide clear instructions for how to do it.

However, i dont really see the conflict. Of course it should not do a iptables -F. All chains have very explicit names, so just cleaning up the ones that belong to openvswitch agent should not cause any issue with other stuff right?

I have not looked in detail on the code, but i suspect that there are other things that have a similar behavior. For ex switching on and off l2_pop and arp_response does not seem to clean up properly either. I could be wrong about this thou. We have a very weird behavior in our cloud right now where pakets are simly just dispearing, really vanishing. ARPs not being responded to. DHCP boot discovers/offers going out on the wrong vxlan, etc, etc. We have no idea whats going on, so we are playing around with different options to see if something can change the behavior. Thats when we noticed this cleanup "issue".

[YAMAMOTO Takashi (yamamoto)]

i don't think it's realistic to behave "nicely" among every kinds of configuration changes.
on the other hand, as iptables is a shared resource, it makes sense to document how our components use it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/911957

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/911957
Committed: https://opendev.org/openstack/neutron/commit/46245c015403c5770d2bd9b6d08f52f89fd6aa40
Submitter: "Zuul (22348)"
Branch: master

commit 46245c015403c5770d2bd9b6d08f52f89fd6aa40
Author: Brian Haley <email address hidden>
Date: Thu Mar 7 14:00:21 2024 -0500

    Add note on iptables cleanup after OVS firewall migration

    Add an item to the instructions on iptables to OVS
    firewall migration that the admin should cleanup
    any stale iptables rules after completion. It is
    out of scope of our documents on how exactly an
    adminstrator might do that.

    Closes-bug: #1864374
    Change-Id: Ie1bf6b82e57a00f61640a131a29d897a9cde4629