[RFE] Add possibility to define default security group rules

[Slawek Kaplonski]

Currently when default security group rule is created for every new project, there are hardcoded 4 rules added to it. Those rules allows:
1. IPv4 egress traffic from port,
2. IPv6 egress traffic from port,
3 IPv4 ingress traffic to port incoming from other ports which are using same security group,
4. IPv6 ingress traffic to port incoming from other ports which are using same security group.

There is couple of issues with that:
1. it is known fact that SG rules with remote_group_id (rule 3. and 4. above) don't scale well e.g. with neutron-openvswitch-agent,
2. Some operators would like to define different rules to be created by default for each new project.

So this RFE propose to add possibility to define for operators (admin user maybe) SG rules which will be added by default for default security group for each project.
To keep backward compatybility with what we have now and what is working like that since many years, by default we may have configure those 4 rules mentioned above as default SG rules but operator (admin user) will have possibility to change it.

I mentioned that it can be defined by operator or admin user as we may implement it as new API which will be available for admins only or e.g. by some special config file (something similar to policy.yaml) and then it can be possible to modify it by clouds operator.

[Miguel Lavalle (minsel)]

Looks good to me. Captures the conversation we had

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-specs/+/857858

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-specs/+/857858
Committed: https://opendev.org/openstack/neutron-specs/commit/d55ea74e5991dddc50212241376fb51516733873
Submitter: "Zuul (22348)"
Branch: master

commit d55ea74e5991dddc50212241376fb51516733873
Author: Slawek Kaplonski <email address hidden>
Date: Thu Sep 15 11:50:50 2022 +0200

    Add spec for the Default SG rules API

    Related-Bug: #1983053
    Change-Id: Ie93c87608191366928b9e37182bc6220cbbd9d7c

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/869554

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/869554
Committed: https://opendev.org/openstack/neutron/commit/d73f75c551c36b15eed43f0b19ed796626cc8cb5
Submitter: "Zuul (22348)"
Branch: master

commit d73f75c551c36b15eed43f0b19ed796626cc8cb5
Author: Slawek Kaplonski <email address hidden>
Date: Mon Jan 9 12:05:28 2023 +0100

    [API] Add API extension and definition for default SG rules

    This patch adds API definition and API extension class for
    security group rules templates API described in the spec [1].
    API definition in this case is very similar to the securitygroup API
    definition and uses same converters and validators which are still in
    Neutron instead of neutron-lib repo. Because of that this new API
    definition is proposed to the neutron repo first and will be rehomed to
    neutron-lib together with security groups API definition later.

    [1] https://specs.openstack.org/openstack/neutron-specs/specs/2023.1/configurable-default-sg-rules.html

    Related-bug: #1983053
    Change-Id: I3aafe1aba406a52bc2b57be5133dee15b8848796

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/883246

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-specs/+/883267

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-specs/+/883268

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/883269

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-specs/+/883295

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/883386

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-specs/+/883481

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/883553

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-specs/+/883295
Committed: https://opendev.org/openstack/neutron-specs/commit/55b8b9e88683a6b0356512c136f6265718ae6a58
Submitter: "Zuul (22348)"
Branch: master

commit 55b8b9e88683a6b0356512c136f6265718ae6a58
Author: Slawek Kaplonski <email address hidden>
Date: Tue May 16 20:02:07 2023 +0200

    Move configurable default SG rules spec to 2023.2 cycle

    This spec was approved originally in the 2023.1 cycle but I didn't had
    time to implement it then.
    So now lets move it to the 2023.2 cycle and implement it finally.

    Related-bug: #1983053
    Change-Id: Id64231e30678207ee36225c18ce72b9928303afb

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-specs/+/883267
Committed: https://opendev.org/openstack/neutron-specs/commit/73f68a1fdae9631621f0c9a64261b3190c62bae9
Submitter: "Zuul (22348)"
Branch: master

commit 73f68a1fdae9631621f0c9a64261b3190c62bae9
Author: Slawek Kaplonski <email address hidden>
Date: Tue May 16 16:07:51 2023 +0200

    Add "used_in_non_default_sg" attribute to the default SG rules API

    This patch adds new parameter to the API of the default SG rules
    templates API. New parameter is called
    "used_in_non_default_security_group" and will be used to mark if rule
    should be used in SGs other than "default" one for the project.

    Related-bug: #1983053
    Change-Id: Ic2fd7b00d4a9de150252eddaffd0409da0925e99

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-specs/+/883268
Committed: https://opendev.org/openstack/neutron-specs/commit/7600a542f724cbb1e4b711e148d247c6713b8d17
Submitter: "Zuul (22348)"
Branch: master

commit 7600a542f724cbb1e4b711e148d247c6713b8d17
Author: Slawek Kaplonski <email address hidden>
Date: Tue May 16 16:12:47 2023 +0200

    Add "remote_address_group_id" attribute to the default SG rules API

    This patch adds new parameter to the API of the default SG rules
    templates API. New parameter is called
    "remote_address_group_id" and can be used to define uuid of the remote
    address group which will be referenced in default rule(s) created for
    each new SG.

    Additionally this patch updates type of the "remote_ip_prefix" field in
    the database. It was set by mistake to "Integer" but should be "String".

    Related-bug: #1983053
    Change-Id: Ieccd6e70bce6be9a16d38b25efc2774ffefe1699

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-specs/+/883481
Committed: https://opendev.org/openstack/neutron-specs/commit/03e11287c7398549c5b8869b54b742c0f921c6ce
Submitter: "Zuul (22348)"
Branch: master

commit 03e11287c7398549c5b8869b54b742c0f921c6ce
Author: Slawek Kaplonski <email address hidden>
Date: Thu May 18 10:39:27 2023 +0200

    Default SG rules - update fields in the API examples

    This patch updates "configurable default SG rules" spec by
    adding "description" field and remove other standard attributes such
    as "created_at", "updated_at" and "revision_number" from the exmaples of
    the API requests and responses.

    Related-Bug: #1983053
    Change-Id: I3fd5f08e691db3ab025a1c64f5105749234ae0c2

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/884475

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/883269
Committed: https://opendev.org/openstack/neutron/commit/a72e97ddff1c2a13cc3dc7636b927bd28bdf2ae3
Submitter: "Zuul (22348)"
Branch: master

commit a72e97ddff1c2a13cc3dc7636b927bd28bdf2ae3
Author: Slawek Kaplonski <email address hidden>
Date: Tue May 16 16:17:58 2023 +0200

    Update api extension for default sg rules API

    This patch adds two new attributes to the default SG rules API:
    * "used_in_non_default_security_group",
    * "remote_address_group_id"

    Those new attributes are descibed in the proposed update to the related
    spec in [1] and [2].

    [1] https://review.opendev.org/c/openstack/neutron-specs/+/883267
    [2] https://review.opendev.org/c/openstack/neutron-specs/+/883268

    Related-bug: #1983053
    Change-Id: Ic3e06460ac8294bfa882991eb678878b238735d7

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/884578

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/883386
Committed: https://opendev.org/openstack/neutron/commit/1b9a16c9567e9fccb48388b38b5a5335e2ff30c9
Submitter: "Zuul (22348)"
Branch: master

commit 1b9a16c9567e9fccb48388b38b5a5335e2ff30c9
Author: Slawek Kaplonski <email address hidden>
Date: Wed May 17 17:00:53 2023 +0200

    Add description field to the security_group_default_rules resource

    This new resource has standard attributes and should expose description
    field in the API.

    Related-bug: #1983053
    Change-Id: Ie2940e6c705e6692eaaf53f11d11b4b62cd0a51e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/884578
Committed: https://opendev.org/openstack/neutron-lib/commit/4b9753de30fca22c59e4836ae2c01fc7ae5a2e37
Submitter: "Zuul (22348)"
Branch: master

commit 4b9753de30fca22c59e4836ae2c01fc7ae5a2e37
Author: Slawek Kaplonski <email address hidden>
Date: Mon May 29 11:12:57 2023 +0200

    [API REF] Add api-ref documentation for default SG rules API

    Related-Bug: #1983053
    Change-Id: I0053337686d49229b44e157977b2607051ad1604

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/891040

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/883553
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/891040
Committed: https://opendev.org/openstack/neutron/commit/49fcd2f515526ed1c0377ad6236c49c7e0d86b2a
Submitter: "Zuul (22348)"
Branch: master

commit 49fcd2f515526ed1c0377ad6236c49c7e0d86b2a
Author: Slawek Kaplonski <email address hidden>
Date: Thu Aug 10 16:33:46 2023 +0200

    Force DB migration script to be run before some fullstack tests

    Db migration is when OpportunisticSqlFixture is setup during the test.
    But it may happen, when 2 tests are run by the same worker that db
    migration will be run only for first test and next one will have
    just empty db schema (especially when tests are run serially, like
    security groups related tests in fullstack CI job).
    To avoid that issue in serially run fullstack tests and to have
    always proper data in the db before test will start, new flag
    FORCE_DB_MIGRATION is added to the OpportunisticDBTestMixin class. It is
    set to False by default so behavior for tests which inherits from this
    class is not changed.
    It's set to True only in the
    BaseSecurityGroupsSameNetworkTest so those fullstack tests will have
    always data in the db at the beginning of the test.

    Related-bug: #1983053
    Change-Id: I1c93f80d6bc19084d30340be5c4b57dbe756a808

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/893097

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/893097
Committed: https://opendev.org/openstack/neutron/commit/78bc33d300dc268d193fd2a895ed26079d441c3d
Submitter: "Zuul (22348)"
Branch: master

commit 78bc33d300dc268d193fd2a895ed26079d441c3d
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 29 19:50:33 2023 +0200

    [Fullstack] Use new DB for each running test

    Until now neutron fullstack tests which were run by the same worker were
    using same DB but after test content of the DB was cleaned.
    This could cause problems e.g. for default security group rules which
    weren't created properly in second test run by the same worker.

    To fix that issue patch [1] was proposed and merged some time ago. But
    this didn't solve the problem so this patch is effectively reverting [1]
    and proposing another solution which will make each fullstack test to
    use own DB and run db migration script.

    As running DB migration before every test makes this jobs to run a bit
    longer than it took before, this patch also increases timeout for the
    fullstack job(s) to 3h (10800 seconds).

    [1] https://review.opendev.org/c/openstack/neutron/+/891040

    Related-bug: #1983053
    Change-Id: Ia261b4c62db9a99ef6eb161acb4609520e45d101

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/884474
Committed: https://opendev.org/openstack/neutron/commit/a4c8392209f7935cc6699c1cf9dc36d483b2f864
Submitter: "Zuul (22348)"
Branch: master

commit a4c8392209f7935cc6699c1cf9dc36d483b2f864
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 26 12:01:31 2023 +0200

    Default SG rules - use new rules templates to create rules for SGs

    Default SG rules created as template in the Neutron DB are now used to
    create security group rules for each new default and non-default SG
    created in Neutron.

    Closes-bug: #1983053
    Change-Id: Iaf27deb955c3844409fcd36239511478e9607a82

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/887664
Committed: https://opendev.org/openstack/neutron/commit/5c2f54ca0364384352035cd1d80acc6e2ffa9f12
Submitter: "Zuul (22348)"
Branch: master

commit 5c2f54ca0364384352035cd1d80acc6e2ffa9f12
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 5 12:09:06 2023 +0200

    Default SG rules template - Update related docs and add release note

    This patch updates docs related to the Security Groups to add info about
    possibility to change default set of rules created in every new security
    group.
    It also adds release note about this new API in Neutron.

    Closes-Bug: #1983053
    Change-Id: I0f6ecc5cf374a0090930e9786834ed7a1be3dc0b

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 23.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/884475
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/883553
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/aa22c9e1bb5bb1a7362dd4973a59a6572dc4732c
Submitter: "Zuul (22348)"
Branch: master

commit aa22c9e1bb5bb1a7362dd4973a59a6572dc4732c
Author: Slawek Kaplonski <email address hidden>
Date: Thu May 18 18:59:26 2023 +0200

    New basic API tests for the default SG rules templates CRUDs

    This patch adds some basic API tests for the new API for default SG
    rules templates. Those new tests are checking if by default SG rules are
    set in the same way as legacy rules which were there since "forever".
    Second test checks basic lifecycle of the SG rule template.

    Depends-On: https://review.opendev.org/c/openstack/neutron/+/883246/

    Related-Bug: #1983053
    Change-Id: I458f54ff6b73e277fe9506e90fa6af44d9c51101

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/884475
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/a1f654aa71b1e8b073a48d7a39923e711b7549f5
Submitter: "Zuul (22348)"
Branch: master

commit a1f654aa71b1e8b073a48d7a39923e711b7549f5
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 26 11:21:27 2023 +0200

    [Default SG rules] Test to check if SG rules are created from template

    This patch adds new test which checks if SG rules actually created
    automatically for new default and non-default SG are matching template
    rules from neutron DB.

    Depends-On: https://review.opendev.org/c/openstack/neutron/+/884474

    Related-bug: #1983053
    Change-Id: Ica0810413bef7f0e3e6dff21f6c9e4cda1945a43