Neutron OVSHybridIptablesFirewallDriver and IptablesFirewallDriver don't enforce Remote address groups

[Robert Breker]

High level description -

The Neutron API allows operators to configure remote address groups [1], however the OVSHybridIptablesFirewallDriver and IptablesFirewallDriver do not implement these remote group restrictions. When configuring security group rules with remote address groups, connections get enabled based on other rule parameters, ignoring the configured remote address group restrictions.
This behaviour undocumented, and may lead to more-open-than-configured network access.

Background -

Remote address groups enable specifying rules that target many CIDRs efficiently. In line with the remote security group support, this should be implemented through the use of hashed ipsets in case of the IptablesFirewallDriver.

Pre-conditions -
* Using OVSHybridIptablesFirewallDriver or IptablesFirewallDriver
* Configured remote Address Groups.

Version -
All OpenStack versions with remote address group support are impacted. We noticed it on 2024.1.

[1] https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/address-group.html

[Robert Breker (rbreker)]

Patch for this issue.

[Brian Haley (brian-haley)]

Interesting, looking at the BP, there was a tempest test added for all firewall drivers, but it didn't fail with openvswitch mech driver and iptables hybrid firewall driver. We need to make sure that test is enabled correctly and catches this.

https://review.opendev.org/q/topic:%22bp/address-groups-in-sg-rules%22

And thanks for the patch, not sure if it also needs a change in _expand_sg_rule_with_remote_ips().

[Lajos Katona (lajos-katona)]

Thanks for reporting, do you plan / are you able to push your patch upstream?
I quickly put on master Neutron and the basic tests (pep8, unit tests) passed.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

[Jeremy Stanley (fungi)]

Based on the description, this sounds like a situation that an attacker wouldn't be able to create, but rather a security feature not working as intended leaving systems exposed to subsequent attacks. Is that an accurate assessment? If so, I don't think we need to work on the fix for this in secret and can switch to our public workflow to get a fix in place and operators notified so they can take action as soon as possible (ideally before 2024.1 is tagged so that we don't release it with this flaw).

I see the mention of 2024.1 being affected, so we're at a minimum going to need a patch for master and a backport to the stable/2024.1 branch. Can someone work out what earlier releases may also be affected?

[Robert Breker (rbreker)]

> [Brian Haley] not sure if it also needs a change in _expand_sg_rule_with_remote_ips().

Indeed. A change to _expand_sg_rule_with_remote_ips() is needed to also fix the issue for enable_ipset=False.
This is now fixed in Enhance-IptablesFirewallDriver-with-remote-address-g-v2.patch

> [Lajos Katona] do you plan / are you able to push your patch upstream?

I'd be happy to give this a try, as soon as the group confirms that no embargo is needed.

> [Jeremy Stanley] Based on the description, this sounds like a situation that an attacker wouldn't be able to create, but rather a security feature not working as intended leaving systems exposed to subsequent attacks. Is that an accurate assessment?

Yes, exactly.
Please let me know, whether I should submit the patch using Gerrit.

> [Jeremy Stanley] Can someone work out what earlier releases may also be affected?

Address groups were first introduced in Wallaby [1], and all releases including and since Wallaby seem affected.

[1] https://releases.openstack.org/wallaby/highlights.html

[Lajos Katona (lajos-katona)]

@Jeremy: remote-adress-group support for security-groups were introduced with this series:
https://review.opendev.org/q/topic:%22bp/address-groups-in-sg-rules%22
and it was done only for OVS fw driver, not for others, so this is a bug for all previous releases.

I just realized that the api-red also miss this field (https://opendev.org/openstack/neutron-lib/src/branch/master/neutron_lib/api/definitions/security_groups_remote_address_group.py )

[Jeremy Stanley (fungi)]

Thanks. Let's work on this in public in that case. Please get fixes for master and stable/2024.1 pushed to Gerrit and reviewing/merging those prioritized so we can have a new RC and not release with this. Separately, we'll need backported fixes at least as far as stable/zed (backports to unmaintained/.* branches are optional at your discretion). Once the necessary backports are proposed I can put together a security advisory for this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/913708

[Robert Breker (rbreker)]

> [Brian Haley] We need to make sure that test is enabled correctly and catches this.

Looks like the scenario test_remote_group_and_remote_address_group() explicitly skips firewall drivers except for openvswitch here;

https://github.com/openstack/neutron-tempest-plugin/blob/bcabf136e59684baf038fd02fc226946c8be63f7/neutron_tempest_plugin/scenario/test_security_groups.py#L788-L790

When we remove this condition, tests should fail, hopefully without much other fallout. I'll propose a test fix in anticipation.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/913850

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/913708
Committed: https://opendev.org/openstack/neutron/commit/5e1188ef38da3f196aadf82a3842fa982c9a0c83
Submitter: "Zuul (22348)"
Branch: master

commit 5e1188ef38da3f196aadf82a3842fa982c9a0c83
Author: Robert Breker <email address hidden>
Date: Sun Mar 17 14:43:50 2024 +0000

    Enhance IptablesFirewallDriver with remote address groups

    This change enhances the IptablesFirewallDriver with support for remote
    address groups. Previously, this feature was only available in the
    OVSFirewallDriver. This commit harmonizes the capabilities across both
    firewall drivers, and by inheritance also to OVSHybridIptablesFirewallDriver.

    Background -
    The Neutron API allows operators to configure remote address groups [1],
    however the OVSHybridIptablesFirewallDriver and IptablesFirewallDriver do
    not implement these remote group restrictions. When configuring security
    group rules with remote address groups, connections get enabled
    based on other rule parameters, ignoring the configured remote address
    group restrictions.
    This behaviour undocumented, and may lead to more-open-than-configured network
    access.

    Closes-Bug: #2058138
    Change-Id: I76b3cb46ee603fa5e829537af41316bb42a6f30f

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/913888

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/neutron/+/913809

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/913809
Committed: https://opendev.org/openstack/neutron/commit/0eccc52f826f21459a285c06a454a3b818d30ca0
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 0eccc52f826f21459a285c06a454a3b818d30ca0
Author: Robert Breker <email address hidden>
Date: Sun Mar 17 14:43:50 2024 +0000

    Enhance IptablesFirewallDriver with remote address groups

    This change enhances the IptablesFirewallDriver with support for remote
    address groups. Previously, this feature was only available in the
    OVSFirewallDriver. This commit harmonizes the capabilities across both
    firewall drivers, and by inheritance also to OVSHybridIptablesFirewallDriver.

    Background -
    The Neutron API allows operators to configure remote address groups [1],
    however the OVSHybridIptablesFirewallDriver and IptablesFirewallDriver do
    not implement these remote group restrictions. When configuring security
    group rules with remote address groups, connections get enabled
    based on other rule parameters, ignoring the configured remote address
    group restrictions.
    This behaviour undocumented, and may lead to more-open-than-configured network
    access.

    Closes-Bug: #2058138
    Change-Id: I76b3cb46ee603fa5e829537af41316bb42a6f30f
    (cherry picked from commit 5e1188ef38da3f196aadf82a3842fa982c9a0c83)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/914217

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/914585

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/914586

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 24.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/914217
Committed: https://opendev.org/openstack/neutron/commit/377e1f6838fcfd5c20056cafaabc7f69992ad2fa
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 377e1f6838fcfd5c20056cafaabc7f69992ad2fa
Author: Robert Breker <email address hidden>
Date: Sun Mar 17 14:43:50 2024 +0000

    Enhance IptablesFirewallDriver with remote address groups

    This change enhances the IptablesFirewallDriver with support for remote
    address groups. Previously, this feature was only available in the
    OVSFirewallDriver. This commit harmonizes the capabilities across both
    firewall drivers, and by inheritance also to OVSHybridIptablesFirewallDriver.

    Background -
    The Neutron API allows operators to configure remote address groups [1],
    however the OVSHybridIptablesFirewallDriver and IptablesFirewallDriver do
    not implement these remote group restrictions. When configuring security
    group rules with remote address groups, connections get enabled
    based on other rule parameters, ignoring the configured remote address
    group restrictions.
    This behaviour undocumented, and may lead to more-open-than-configured network
    access.

    Closes-Bug: #2058138
    Change-Id: I76b3cb46ee603fa5e829537af41316bb42a6f30f
    (cherry picked from commit 5e1188ef38da3f196aadf82a3842fa982c9a0c83)
    (cherry picked from commit 0eccc52f826f21459a285c06a454a3b818d30ca0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/914585
Committed: https://opendev.org/openstack/neutron/commit/872c6ad2cd2e80933f1bf8b8ad6a0e0c1a0169e6
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 872c6ad2cd2e80933f1bf8b8ad6a0e0c1a0169e6
Author: Robert Breker <email address hidden>
Date: Sun Mar 17 14:43:50 2024 +0000

    Enhance IptablesFirewallDriver with remote address groups

    This change enhances the IptablesFirewallDriver with support for remote
    address groups. Previously, this feature was only available in the
    OVSFirewallDriver. This commit harmonizes the capabilities across both
    firewall drivers, and by inheritance also to OVSHybridIptablesFirewallDriver.

    Background -
    The Neutron API allows operators to configure remote address groups [1],
    however the OVSHybridIptablesFirewallDriver and IptablesFirewallDriver do
    not implement these remote group restrictions. When configuring security
    group rules with remote address groups, connections get enabled
    based on other rule parameters, ignoring the configured remote address
    group restrictions.
    This behaviour undocumented, and may lead to more-open-than-configured network
    access.

    Closes-Bug: #2058138
    Change-Id: I76b3cb46ee603fa5e829537af41316bb42a6f30f
    (cherry picked from commit 5e1188ef38da3f196aadf82a3842fa982c9a0c83)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/914586
Committed: https://opendev.org/openstack/neutron/commit/ca25eb96f16dbe5ff1ca4446534f9a1d12fa7035
Submitter: "Zuul (22348)"
Branch: stable/zed

commit ca25eb96f16dbe5ff1ca4446534f9a1d12fa7035
Author: Robert Breker <email address hidden>
Date: Sun Mar 17 14:43:50 2024 +0000

    Enhance IptablesFirewallDriver with remote address groups

    This change enhances the IptablesFirewallDriver with support for remote
    address groups. Previously, this feature was only available in the
    OVSFirewallDriver. This commit harmonizes the capabilities across both
    firewall drivers, and by inheritance also to OVSHybridIptablesFirewallDriver.

    Background -
    The Neutron API allows operators to configure remote address groups [1],
    however the OVSHybridIptablesFirewallDriver and IptablesFirewallDriver do
    not implement these remote group restrictions. When configuring security
    group rules with remote address groups, connections get enabled
    based on other rule parameters, ignoring the configured remote address
    group restrictions.
    This behaviour undocumented, and may lead to more-open-than-configured network
    access.

    Closes-Bug: #2058138
    Change-Id: I76b3cb46ee603fa5e829537af41316bb42a6f30f
    (cherry picked from commit 5e1188ef38da3f196aadf82a3842fa982c9a0c83)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/913888
Committed: https://opendev.org/openstack/neutron-lib/commit/a6e32d9d5c2942dd3fffd04fb05d6e8091d1169a
Submitter: "Zuul (22348)"
Branch: master

commit a6e32d9d5c2942dd3fffd04fb05d6e8091d1169a
Author: elajkat <email address hidden>
Date: Thu Mar 21 10:44:54 2024 +0100

    api-ref: add api-ref for remote_address_group in sg rules

    Related-Bug: #2058138
    Change-Id: I067c53c31a60336b8852cdf26a404b15cc0ccef8

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.2.1 release.