OVS conjunctive flows are not cleaned up after remote group member ips deleted

[Hang Yang]

Running with the current Neutron master and OVS firewall agent in devstack all-in-one, when creating a security group rule with a remote-group for an active VM, the conjunctive flows that match the remote-group's member IPs are created. But when deleting the remote-group's member IPs(e.g: unset fixed-ips of the port associated with the remote-group), the deleted IP's conjunctive flows are not cleaned up in OVS.

Detailed steps to reproduce in devstack: http://paste.openstack.org/show/800820/

[Hang Yang (hangyang)]

The issue was discovered when I was working on the remote-address-group feature for OVS. Removing IPs from remote-group and remote-address-group cannot cleanup the IPs' conjunctive (1/2) flows. I made a change in my address group patch which used ANY cookie to clean IP flows, then the cleanup worked for both remote-group and remote-address-group: https://review.opendev.org/c/openstack/neutron/+/757650/9..10/neutron/agent/linux/openvswitch_firewall/firewall.py#1634

I want to bring up some discussion about if we can confirm it is a bug and either my change is the right way to fix it.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Reviewing the pasted information [1], seems to be a legit bug: the port fixed IP should be deleted from the OF tables when deleted.

[1]http://paste.openstack.org/show/800820/

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

[Jeremy Stanley (fungi)]

I'm trying to think through the risks from this to determine whether it warrants an advisory, and whether it's severe enough to need to be discussed/fixed privately under embargo. So far the worst exploit scenario I can conceive is that some host you've previously allowed connections from in your security groups is found to be compromised and you want to cease allowing access from it, so you remove it from the security group but any already established connections/flows are not terminated automatically and so the system continues to have access when you think you've locked it out.

Unless I'm missing something, while this could certainly cause serious security problems during an incident, I don't see opportunities to directly exploit this bug, making it a likely class D report per our taxonomy (security-relevant bug worth fixing, but doesn't need a coordinated disclosure or widely-distributed advisory and can be fixed in public following our normal development process): https://security.openstack.org/vmt-process.html#incident-report-taxonomy

Unless there are disagreements on this assessment, I'd like to switch the report to public state on Monday (December 14).

[Miguel Lavalle (minsel)]

I am able to reproduce the bug in my allinone devstack environment with Ubuntu 20.04 and master branch. To address this issue Hang has proposed this https://review.opendev.org/c/openstack/neutron/+/757650/10/neutron/agent/linux/openvswitch_firewall/firewall.py#1634. With this, the problem reported in this bug is fixed. We wonder, though, if it might be creating undesirable side effects. Thoughts?

[Jeremy Stanley (fungi)]

With there being a fix already public which hints at the security impact, I feel much more strongly that we should go ahead and make the bug report public so reviewers can have a more complete context.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello @Miguel:

Is it possible to provide a patch just implementing this fix?

Regards.

[Slawek Kaplonski (slaweq)]

IMO this can be public. It seems like serious bug but I don't think there is any way to use it to attack someone's else VMs.

[Jeremy Stanley (fungi)]

Thanks, given the circumstances with fixes already being discussed in public and the low risk of this being directly exploited, I've gone ahead and made the report public now and tagged it as security-related. I've also set our coordinated advisory task to "won't fix" indicating that I think there isn't an obvious need to publish a specific security advisory about it, but am happy to revisit that decision if more pressing exploit scenarios can be presented for it.

[Hang Yang (hangyang)]

Thanks for checking this issue. I have proposed a separate fix patch:
https://review.opendev.org/c/openstack/neutron/+/766775

[Hang Yang (hangyang)]

Fix released and backported:
Master: https://review.opendev.org/c/openstack/neutron/+/766775
Victoria: https://review.opendev.org/c/openstack/neutron/+/767676
Ussuri: https://review.opendev.org/c/openstack/neutron/+/767677
Train: https://review.opendev.org/c/openstack/neutron/+/767678

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.3.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.1.0 release.

[Corey Bryant (corey.bryant)]

This is fixed in the following bugs for Ubuntu:
neutron 17.1.0 (groovy/victoria): https://bugs.launchpad.net/cloud-archive/+bug/1915785
neutron 16.3.0 (focal/ussuri): https://bugs.launchpad.net/cloud-archive/+bug/1915786
neutron (bionic/train): fix released

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/776454
Committed: https://opendev.org/openstack/neutron/commit/480ede535a27caf2bbc82cecfdfbc25cc6d61e39
Submitter: "Zuul (22348)"
Branch: stable/rocky

commit 480ede535a27caf2bbc82cecfdfbc25cc6d61e39
Author: Hang Yang <email address hidden>
Date: Fri Dec 11 12:16:05 2020 -0600

    Fix OVS conjunctive IP flows cleanup

     Currently when deleting a remote-group's member IPs, the deleted IPs'
     conjunctive flows are not cleaned up in OF tables. This is because
     the conjunctive flows' cookies don't match with the OVSBridge default
     cookie used by the delete flow method. This patch fixed the issue by
     using an ANY cookie that can always match with the cookies of the
     conjunctive flows.

    Conflicts:
        neutron/agent/linux/openvswitch_firewall/firewall.py

    Change-Id: I74916acf8311989dca267f23261ec4cf449a6abf
    Closes-Bug: 1907491
    (cherry picked from commit f4b64e519cdb9fd9c5046f21bc9f325341fd367f)
    (cherry picked from commit 03f0a832a82f35286417e56d6071cfefe823a65c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron stein-eol release.