OVS native firewall driver does not work

[Radosław Piliszek]

A lot of such entries are being generated by neutron-ovs-agent when ovs native fireall driver is being used:

2020-03-14 15:34:39.761 6 ERROR neutron.agent.linux.utils [req-b4ecdf6e-d1b8-4cf3-bd57-1141b0834568 - - - - -] Exit code: 1; Stdin: table=73,dl_dst=fa:16:3e:3a:7b:71,reg6=1,cookie=7298701011038913443/-1
2020-03-14 15:34:39.762 6 ERROR neutron.agent.common.ovs_lib [req-b4ecdf6e-d1b8-4cf3-bd57-1141b0834568 - - - - -] Unable to execute ['ovs-ofctl', 'del-flows', '-O', 'OpenFlow10', 'br-int', '--bundle', '-']. Exception: Exit code: 1; Stdin: table=73,dl_dst=fa:16:3e:3a:7b:71,reg6=1,cookie=7298701011038913443/-1

The reason is neutron-ovs-agent still calls ovs-ofctl behind the scenes. This path is deprecated by Neutron for removal in Victoria but currently ovs native firewall driver relies on it.

The bug has been caused by the /run mount removal. See: https://bugs.launchpad.net/kolla-ansible/+bug/1861792
/run/openvswitch must still be mounted in neutron-ovs-agent.

Triage: https://review.opendev.org/713094

The observed effect is that security group rules are silently (except for non-fatal error messages in logs) not applied and all the traffic is accepted.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/713129

[Radosław Piliszek (yoctozepto)]

Note this is not k-a default firewall driver, hence why only 'high', not 'critical'. Also why CI gave us no clue. OTOH, as seen in triage, testing it would still result in green SUCCESS on the outside. This would only have been caught if we both ran this scenario and failed on ERRORs (too harsh for now unless more filtered like CRITICALs). I wonder if running tempest against this breakage would have caught it as well or not.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/713129
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=cf918fbc294be35f662d6b2851f3cdc00bc2b1bb
Submitter: Zuul
Branch: master

commit cf918fbc294be35f662d6b2851f3cdc00bc2b1bb
Author: Radosław Piliszek <email address hidden>
Date: Sun Mar 15 11:26:25 2020 +0100

    Fix native openvswitch firewall driver in neutron-openvswitch-agent

    ovs-ofctl is still being run by neutron-openvswitch-agent.
    Potential removal is scheduled for Victoria.
    Until then, we have to mount /run/openvswitch in there.

    Change-Id: Ia73b5665cece523bb822f6a223335f6fae94fb6a
    Closes-bug: #1867506

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/713378

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/713488

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/713490

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/713378
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=626232fa4ecb263518213c363935277b1f9cd792
Submitter: Zuul
Branch: stable/train

commit 626232fa4ecb263518213c363935277b1f9cd792
Author: Radosław Piliszek <email address hidden>
Date: Sun Mar 15 11:26:25 2020 +0100

    Fix native openvswitch firewall driver in neutron-openvswitch-agent

    ovs-ofctl is still being run by neutron-openvswitch-agent.
    Potential removal is scheduled for Victoria.
    Until then, we have to mount /run/openvswitch in there.

    Change-Id: Ia73b5665cece523bb822f6a223335f6fae94fb6a
    Closes-bug: #1867506
    (cherry picked from commit cf918fbc294be35f662d6b2851f3cdc00bc2b1bb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/713488
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=3eb908272d23e8108ec543d889c565357b8604ca
Submitter: Zuul
Branch: stable/stein

commit 3eb908272d23e8108ec543d889c565357b8604ca
Author: Radosław Piliszek <email address hidden>
Date: Sun Mar 15 11:26:25 2020 +0100

    Fix native openvswitch firewall driver in neutron-openvswitch-agent

    ovs-ofctl is still being run by neutron-openvswitch-agent.
    Potential removal is scheduled for Victoria.
    Until then, we have to mount /run/openvswitch in there.

    Change-Id: Ia73b5665cece523bb822f6a223335f6fae94fb6a
    Closes-bug: #1867506
    (cherry picked from commit cf918fbc294be35f662d6b2851f3cdc00bc2b1bb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/713490
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=3ecc2f73707388924b09e88c6872ec47cf207a78
Submitter: Zuul
Branch: stable/rocky

commit 3ecc2f73707388924b09e88c6872ec47cf207a78
Author: Radosław Piliszek <email address hidden>
Date: Sun Mar 15 11:26:25 2020 +0100

    Fix native openvswitch firewall driver in neutron-openvswitch-agent

    ovs-ofctl is still being run by neutron-openvswitch-agent.
    Potential removal is scheduled for Victoria.
    Until then, we have to mount /run/openvswitch in there.

    Change-Id: Ia73b5665cece523bb822f6a223335f6fae94fb6a
    Closes-bug: #1867506
    (cherry picked from commit cf918fbc294be35f662d6b2851f3cdc00bc2b1bb)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.2.1 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/716174

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/716175

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/716178

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/716179

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716174
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=c033ddca082618062bdd3abe4b4e7ed111063cbd
Submitter: Zuul
Branch: master

commit c033ddca082618062bdd3abe4b4e7ed111063cbd
Author: Radosław Piliszek <email address hidden>
Date: Tue Mar 31 09:01:02 2020 +0200

    Fix ovs fw driver for the other ovs agent

    In [1] only neutron-openvswitch-agent was fixed and not xenapi.
    That merged in Ussuri and went cleanly into Train.
    In Stein and Rocky, the backport was not clean and
    accidentally fixed xenapi instead of the regular one.

    Neither the original bug nor its incomplete fix were released,
    except for Rocky. :-(
    Hence this patch also removes the confusing reno instead of
    adding a new one.

    [1] https://review.opendev.org/713129

    Change-Id: I331417c8d61ba6f180bcafa943be697418326645
    Closes-bug: #1869832
    Related-bug: #1867506

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716178
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=5da311c69d0e877f16d4fae1b2a5e2897f729ada
Submitter: Zuul
Branch: stable/stein

commit 5da311c69d0e877f16d4fae1b2a5e2897f729ada
Author: Radosław Piliszek <email address hidden>
Date: Tue Mar 31 09:01:02 2020 +0200

    Fix ovs fw driver for the other ovs agent

    In [1] only neutron-openvswitch-agent was fixed and not xenapi.
    That merged in Ussuri and went cleanly into Train.
    In Stein and Rocky, the backport was not clean and
    accidentally fixed xenapi instead of the regular one.

    Neither the original bug nor its incomplete fix were released,
    except for Rocky. :-(
    Hence this patch also removes the confusing reno instead of
    adding a new one.

    [1] https://review.opendev.org/713129

    Change-Id: I331417c8d61ba6f180bcafa943be697418326645
    Closes-bug: #1869832
    Related-bug: #1867506
    (cherry picked from commit c033ddca082618062bdd3abe4b4e7ed111063cbd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716175
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=1927ba28eee49dbe564c9f5d900a25ec97eaa875
Submitter: Zuul
Branch: stable/train

commit 1927ba28eee49dbe564c9f5d900a25ec97eaa875
Author: Radosław Piliszek <email address hidden>
Date: Tue Mar 31 09:01:02 2020 +0200

    Fix ovs fw driver for the other ovs agent

    In [1] only neutron-openvswitch-agent was fixed and not xenapi.
    That merged in Ussuri and went cleanly into Train.
    In Stein and Rocky, the backport was not clean and
    accidentally fixed xenapi instead of the regular one.

    Neither the original bug nor its incomplete fix were released,
    except for Rocky. :-(
    Hence this patch also removes the confusing reno instead of
    adding a new one.

    [1] https://review.opendev.org/713129

    Change-Id: I331417c8d61ba6f180bcafa943be697418326645
    Closes-bug: #1869832
    Related-bug: #1867506
    (cherry picked from commit c033ddca082618062bdd3abe4b4e7ed111063cbd)