Neutron should clean up ACLs in OVN NB DB when a remote security group is deleted

Bug #1983600 reported by Miro Tomaska
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Low
Rodolfo Alonso

Bug Description

Neutron does not clean up ACLs in OVN NB DB when a remote security group is deleted in Neutron. This causes some warning in OVN logs. This issue does not impact functionality but it would be great if we could clean this up and make those logs go away. I think this BZ can be marked as low priority and low hanging fruit.

Reproducing steps:

1. A security group rule which has a rule that uses a remote security group

openstack security group rule list 44038bce-38fd-4219-ae22-bafaad9bbde9
+--------------------------------------+-------------+----------+------------+--------------------------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+----------+------------+--------------------------------------+
| 1155ffc7-9a99-41c7-bd65-fac33b719973 | icmp | None | | 4b0f872b-22ae-4ac2-b54e-ea1678a88dd5 |
| 8e4cdd63-c3e8-411f-8d72-c5cced5d4fc8 | None | None | | None |
| b72b29df-d5f1-4815-92ae-71b1cdc35f97 | None | None | | None |
+--------------------------------------+-------------+----------+------------+--------------------------------------+

Here is a view in ovn nb db

docker exec -it ovn-dbs-bundle-docker-1 ovn-nbctl acl-list pg_44038bce_38fd_4219_ae22_bafaad9bbde9
from-lport 1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4) allow-related
from-lport 1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip6) allow-related
  to-lport 1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related

2. now remove the remote security group and check the rules in the first security group

openstack security group delete 4b0f872b-22ae-4ac2-b54e-ea1678a88dd5

openstack security group rule list 44038bce-38fd-4219-ae22-bafaad9bbde9
+--------------------------------------+-------------+----------+------------+-----------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+----------+------------+-----------------------+
| 8e4cdd63-c3e8-411f-8d72-c5cced5d4fc8 | None | None | | None |
| b72b29df-d5f1-4815-92ae-71b1cdc35f97 | None | None | | None |
+--------------------------------------+-------------+----------+------------+-----------------------+

So from neutron the security group rule is removed

3. check the acl in ovn again and see they are still there

docker exec -it ovn-dbs-bundle-docker-1 ovn-nbctl acl-list pg_44038bce_38fd_4219_ae22_bafaad9bbde9
from-lport 1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4) allow-related
from-lport 1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip6) allow-related
  to-lport 1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related

this causes warnings to be generated in ovn

2020-11-17T14:04:59Z|00708|lflow|WARN|Dropped 1 log messages in last 917 seconds (most recently, 917 seconds ago) due to excessive rate
2020-11-17T14:04:59Z|00709|lflow|WARN|error parsing match "((ct.new && !ct.est) || (!ct.new && ct.est && !ct.rpl && ct_label.blocked == 1)) && (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4)": Syntax error at `$pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4' expecting address set name.

What that is happening:
Only security groups that are not in use by other ports can be deleted. If a security group referenced as remote by other group is removed, the security group rule is deleted in DB because of the references in DB tables. It means there is no code triggered in the Neutron code. The corresponding port group is deleted in OVN but OVN doesn't have any on delete cascade functionality like SQL has. It means ACL remains in the OVN DB using port group that no longer exists. It has no impact on the traffic because the deleted security group was not in use.

Miro Tomaska (mtomaska)
description: updated
Changed in neutron:
importance: Undecided → Low
status: New → Confirmed
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

That's right: when the remote SG is deleted, the corresponding SG rules, in the Neutron DB, are deleted; but the OVN ACLs don't. This is because the Neutron DB registers "SecurityGroupRule" are deleted by the DB engine because "remote_group_id" is a foreign key that is removed when the reference is:

class SecurityGroupRule(standard_attr.HasStandardAttributes, model_base.BASEV2,
                        model_base.HasId, model_base.HasProject):
    """Represents a v2 neutron security group rule."""
    remote_group_id = sa.Column(sa.String(36),
                                sa.ForeignKey("securitygroups.id",
                                              ondelete="CASCADE"),
                                nullable=True)

And there is another problem: the method "OVNMechanismDriver._delete_security_group" is called AFTER_DELETE. That means both the SG and the related SG rules have been removed. We'll probably need to capture this event before the SG deletion.

Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/852742

Changed in neutron:
status: Confirmed → In Progress
tags: added: backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/852742
Committed: https://opendev.org/openstack/neutron/commit/1957353ef92ec11766a2ce3d29d51afa8d7fdb3e
Submitter: "Zuul (22348)"
Branch: master

commit 1957353ef92ec11766a2ce3d29d51afa8d7fdb3e
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Aug 9 20:10:42 2022 +0200

    [OVN] Remove ACLs with remote SG during deletion of SG

    When a security group is removed, the rules having this security
    group as remote are removed. But the OVN ACL registers related
    to those rules are not removed. This patch catches the security
    group deletion precommit event to perform this cleanup.

    Closes-Bug: #1983600

    Change-Id: I6bb84cb748a2f80f2ff640ceeb3223413f7e92c7

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/854558

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/854559

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/854561

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/854559
Committed: https://opendev.org/openstack/neutron/commit/3f0ea85becbc65f0ffa6a40efcc87c27cb0ca615
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 3f0ea85becbc65f0ffa6a40efcc87c27cb0ca615
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Aug 9 20:10:42 2022 +0200

    [OVN] Remove ACLs with remote SG during deletion of SG

    When a security group is removed, the rules having this security
    group as remote are removed. But the OVN ACL registers related
    to those rules are not removed. This patch catches the security
    group deletion precommit event to perform this cleanup.

    Closes-Bug: #1983600

    Change-Id: I6bb84cb748a2f80f2ff640ceeb3223413f7e92c7
    (cherry picked from commit 1957353ef92ec11766a2ce3d29d51afa8d7fdb3e)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/854561
Committed: https://opendev.org/openstack/neutron/commit/fe645cefe6cdce08b181a63acced7a47281feecb
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit fe645cefe6cdce08b181a63acced7a47281feecb
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Aug 9 20:10:42 2022 +0200

    [OVN] Remove ACLs with remote SG during deletion of SG

    When a security group is removed, the rules having this security
    group as remote are removed. But the OVN ACL registers related
    to those rules are not removed. This patch catches the security
    group deletion precommit event to perform this cleanup.

    Closes-Bug: #1983600

    Conflicts:
        neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py
        neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py

    Change-Id: I6bb84cb748a2f80f2ff640ceeb3223413f7e92c7
    (cherry picked from commit 1957353ef92ec11766a2ce3d29d51afa8d7fdb3e)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/854558
Committed: https://opendev.org/openstack/neutron/commit/0cccea36ffb0658cce03a16f65d4024c2de27c11
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 0cccea36ffb0658cce03a16f65d4024c2de27c11
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Aug 9 20:10:42 2022 +0200

    [OVN] Remove ACLs with remote SG during deletion of SG

    When a security group is removed, the rules having this security
    group as remote are removed. But the OVN ACL registers related
    to those rules are not removed. This patch catches the security
    group deletion precommit event to perform this cleanup.

    Closes-Bug: #1983600

    Change-Id: I6bb84cb748a2f80f2ff640ceeb3223413f7e92c7
    (cherry picked from commit 1957353ef92ec11766a2ce3d29d51afa8d7fdb3e)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 21.0.0.0rc1

This issue was fixed in the openstack/neutron 21.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.6.0

This issue was fixed in the openstack/neutron 18.6.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.5.0

This issue was fixed in the openstack/neutron 19.5.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.3.0

This issue was fixed in the openstack/neutron 20.3.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.