Neutron should not create security group with project==None

[Dr. Jens Harbott]

When a non-admin user tries to list security groups for project_id "None", Neutron creates a default security group for that project and returns an empty list to the caller.

To reproduce:

openstack --os-cloud devstack security group list --project None
openstack --os-cloud devstack-admin security group list

The API call that is made is essentially

GET /networking/v2.0/security-groups?project_id=None

The expected result would be an authorization failure, since normal users should not be allowed to list security groups for other projects.

[Lajos Katona (lajos-katona)]

Opensearch link:
https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-14d,to:now))&_a=(columns:!(build_name,build_status),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22Cannot%20find%20Port_Group%20with%20name%3D%22'),sort:!())

[Dr. Jens Harbott (j-harbott)]

Note that the opensearch link only shows the failures that occur when two tasks try to delete that same security group in parallel. This used to succeed earlier (most of the time?), but is failing most of the time since a couple of days. The issue in this bug seems to have been existing for quite some time and is not directly related.

See https://review.opendev.org/c/openstack/tempest/+/854973 for a patch that will prevent Tempest from triggering the issue, but a proper fix in Neutron would still be warranted IMO.

[Brian Haley (brian-haley)]

It's actually worse that just the None case Jens mentioned, I can use any random string and create a default security group for it as 'None' is what is actually being used for the project ID, not the python None.

And I verified that quotas will not catch this, I was able to create default SGs for more than 10 "project IDs" successfully.

I started looking at the code but it's end-of-day here so will look further tomorrow.

But I'll change this to Critical to get some eyes on it.

[Jeremy Stanley (fungi)]

After a brief discussion with Brian Haley, it seems like this represents a minor to moderate security risk: an authenticated user can create an unbounded number of empty security groups because they're not limited by the project quota (since these are not created for their project). I'm adding an incomplete security advisory task for now to indicate that the VMT is following this, and subscribing the neutron-coresec team for added visibility. When a fix is devised, if it's a backportable solution, we can further discuss whether this is severe enough to warrant broad distribution of an advisory statement as well.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/855580

[Dr. Jens Harbott (j-harbott)]

Oh, nice. This even works when supplying valid project IDs of different projects. The result isn't different though then what would happen when the project does their own initial contact with Neutron, so I think there is no additional exploit hidden there.

Note however that in the default configuration, the created security groups will not be empty, but contain four rules (allow outgoing and remote=self for both IPv4 and IPv6), that will slightly increase the resource usage impact.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/855580
Committed: https://opendev.org/openstack/neutron/commit/01fc2b9195f999df4d810df4ee63f77ecbc81f7e
Submitter: "Zuul (22348)"
Branch: master

commit 01fc2b9195f999df4d810df4ee63f77ecbc81f7e
Author: Brian Haley <email address hidden>
Date: Thu Sep 1 21:13:44 2022 -0400

    Do not allow a tenant to create a default SG for another one

    The attempt to list security groups for a project, or any
    random string, can create a default SG for it. Only allow if
    privileges support it.

    Closes-bug: #1988026

    Change-Id: Ieef7011f48cd2188d4254ff16d90a6465bbabfe3

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/858469

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/858617

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/858618

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/858619

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/858620

[Nick Tait (nickthetait)]

So far I agree with a moderate impact assessment. Here is a recap as I understand the situation:
Any non-admin user can cause new security group to be created by listing the security groups of an invalid project_id. This new security group includes 4 default rules which will consume memory and CPU cycles. This action can be repeated an ulimited number of times as there is no quota limit, which could eventually cause a denial of service.

Would it be appropriate for Red Hat to assign this a CVE?

[Jeremy Stanley (fungi)]

Nick: From a project standpoint, we can't prevent anyone from assigning a CVE to any particular bug, but take no position on whether someone should. All we ask is that you let us know (by updating this bug report) what CVE number gets assigned so that if we or someone else later decide it needs a CVE we don't end up with duplicate assignments. The main question to the project is whether this warrants a broadly circulated advisory. Resource consumption bugs only reachable by authenticated users tend to straddle the line on practical exploit vs security hardening opportunity.

If there are other known ways for authenticated users to create resources which can fill up databases as quickly (regardless of whether they would "get billed" for them: someone who is malicious enough to do this likely has no intent of paying a bill anyway), then publishing an advisory would send a misleading signal that we don't expect authenticated users to be able to consume resources. The speed at which they can fill a reasonable sized database in a production deployment and whether or not this activity is likely to be detected by typically employed API rate limiters or resource monitoring systems also informs the decision on whether this rises to the level of a security advisory.

Neutron maintainers: What's your take on these factors?

[Dr. Jens Harbott (j-harbott)]

As mentioned earlier, the number of Neutron resources a user/project can create is usually stricly limited by quotas, the default limit is 10 security groups and 100 rules in them [0]. Because of this, it is also unlikely that deployments use additional resource monitoring systems for this, since they can expect the quotas to be respected.

Also, since the trigger is just a GET query, not a PUT/POST, creating an empty response, I'd assume API rate limiters to also usually not be too strict about this, if they are at all.

Lastly, the main DoS vector likely is not the database size itself filling up, but performance issues in the lookup of security groups and rules, though I have no idea how fast this would happen and how severe the impact might be. Maybe some testing on this might be needed. But I would consider a plausible result could be that other rules fail to get applied in the expected way.

All in all I think that an advisory could be warranted, but would also be very interested to hear other Neutron maintainers' opinions.

[0] https://docs.openstack.org/neutron/latest/admin/ops-quotas.html#basic-quota-configuration

[Nick Tait (nickthetait)]

I have reserved CVE-2022-3277. Sounds like some exploratory testing is needed to get a better handle on impact.

[Brian Haley (brian-haley)]

Since I'm the Neutron maintainer that triaged and fixed this I'm a little biased, but will just repeat some of the above for clarity:

1. You must be an authenticated user of the cloud to make the request
2. This is a GET request that in essence has the effect of a POST because it creates a resource
3. The resource created does not affect the quota of the user making the request
4. The DoS vector is going to consume API and DB resources, albeit a "small" amount
5. There is no action taken by the system - no agent will be triggered to instantiate the fake
   security groups

I think if #5 was true it would warrant more cause for alarm, since something like the OVS agent doing a SG recalculation could consume more CPU resources.

The severity of the bug (critical) was more an indication fixing it should be prioritized over other bugs for the week, especially once I realized it got around the quota limit.

So IMHO it's at most a security advisory, but am open to other maintainer's opinions.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858469
Committed: https://opendev.org/openstack/neutron/commit/fd7fb0e9d8c602380f54975367d935ab69e10c05
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit fd7fb0e9d8c602380f54975367d935ab69e10c05
Author: Brian Haley <email address hidden>
Date: Thu Sep 1 21:13:44 2022 -0400

    Do not allow a tenant to create a default SG for another one

    The attempt to list security groups for a project, or any
    random string, can create a default SG for it. Only allow if
    privileges support it.

    Closes-bug: #1988026

    Change-Id: Ieef7011f48cd2188d4254ff16d90a6465bbabfe3
    (cherry picked from commit 01fc2b9195f999df4d810df4ee63f77ecbc81f7e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858619
Committed: https://opendev.org/openstack/neutron/commit/d0e1b54fb1de932b2b30ab4269cf5789632df476
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit d0e1b54fb1de932b2b30ab4269cf5789632df476
Author: Brian Haley <email address hidden>
Date: Thu Sep 1 21:13:44 2022 -0400

    Do not allow a tenant to create a default SG for another one

    The attempt to list security groups for a project, or any
    random string, can create a default SG for it. Only allow if
    privileges support it.

    Closes-bug: #1988026

    Change-Id: Ieef7011f48cd2188d4254ff16d90a6465bbabfe3
    (cherry picked from commit 01fc2b9195f999df4d810df4ee63f77ecbc81f7e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858618
Committed: https://opendev.org/openstack/neutron/commit/733ef4f2d8c2a3734c360d1c1dd3a6fcd600cb8c
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 733ef4f2d8c2a3734c360d1c1dd3a6fcd600cb8c
Author: Brian Haley <email address hidden>
Date: Thu Sep 1 21:13:44 2022 -0400

    Do not allow a tenant to create a default SG for another one

    The attempt to list security groups for a project, or any
    random string, can create a default SG for it. Only allow if
    privileges support it.

    Closes-bug: #1988026

    Change-Id: Ieef7011f48cd2188d4254ff16d90a6465bbabfe3
    (cherry picked from commit 01fc2b9195f999df4d810df4ee63f77ecbc81f7e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858620
Committed: https://opendev.org/openstack/neutron/commit/cbeee87fa44cd200d4997e02042098460167dce1
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit cbeee87fa44cd200d4997e02042098460167dce1
Author: Brian Haley <email address hidden>
Date: Thu Sep 1 21:13:44 2022 -0400

    Do not allow a tenant to create a default SG for another one

    The attempt to list security groups for a project, or any
    random string, can create a default SG for it. Only allow if
    privileges support it.

    Closes-bug: #1988026

    Change-Id: Ieef7011f48cd2188d4254ff16d90a6465bbabfe3
    (cherry picked from commit 01fc2b9195f999df4d810df4ee63f77ecbc81f7e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858617
Committed: https://opendev.org/openstack/neutron/commit/717e3e09556f1fb9a7a420863746fa785eb6c316
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 717e3e09556f1fb9a7a420863746fa785eb6c316
Author: Brian Haley <email address hidden>
Date: Thu Sep 1 21:13:44 2022 -0400

    Do not allow a tenant to create a default SG for another one

    The attempt to list security groups for a project, or any
    random string, can create a default SG for it. Only allow if
    privileges support it.

    Closes-bug: #1988026

    Change-Id: Ieef7011f48cd2188d4254ff16d90a6465bbabfe3
    (cherry picked from commit 01fc2b9195f999df4d810df4ee63f77ecbc81f7e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron victoria-eom release.