| 2022-08-04 15:30:30 |
Miro Tomaska |
bug |
|
|
added bug |
| 2022-08-04 15:32:35 |
Miro Tomaska |
description |
Neutron does not clean up ACLs in OVN NB DB when a remote security group is deleted in Neutron. This causes some warning in OVN logs. This issue does not impact functionality but it would be great if we could clean this up and make those logs go away.
Reproducing steps:
1. A security group rule which has a rule that uses a remote security group
openstack security group rule list 44038bce-38fd-4219-ae22-bafaad9bbde9
+--------------------------------------+-------------+----------+------------+--------------------------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+----------+------------+--------------------------------------+
| 1155ffc7-9a99-41c7-bd65-fac33b719973 | icmp | None | | 4b0f872b-22ae-4ac2-b54e-ea1678a88dd5 |
| 8e4cdd63-c3e8-411f-8d72-c5cced5d4fc8 | None | None | | None |
| b72b29df-d5f1-4815-92ae-71b1cdc35f97 | None | None | | None |
+--------------------------------------+-------------+----------+------------+--------------------------------------+
Here is a view in ovn nb db
docker exec -it ovn-dbs-bundle-docker-1 ovn-nbctl acl-list pg_44038bce_38fd_4219_ae22_bafaad9bbde9
from-lport 1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4) allow-related
from-lport 1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip6) allow-related
to-lport 1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related
2. now remove the remote security group and check the rules in the first security group
openstack security group delete 4b0f872b-22ae-4ac2-b54e-ea1678a88dd5
openstack security group rule list 44038bce-38fd-4219-ae22-bafaad9bbde9
+--------------------------------------+-------------+----------+------------+-----------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+----------+------------+-----------------------+
| 8e4cdd63-c3e8-411f-8d72-c5cced5d4fc8 | None | None | | None |
| b72b29df-d5f1-4815-92ae-71b1cdc35f97 | None | None | | None |
+--------------------------------------+-------------+----------+------------+-----------------------+
So from neutron the security group rule is removed
3. check the acl in ovn again and see they are still there
docker exec -it ovn-dbs-bundle-docker-1 ovn-nbctl acl-list pg_44038bce_38fd_4219_ae22_bafaad9bbde9
from-lport 1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4) allow-related
from-lport 1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip6) allow-related
to-lport 1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related
this causes warnings to be generated in ovn
2020-11-17T14:04:59Z|00708|lflow|WARN|Dropped 1 log messages in last 917 seconds (most recently, 917 seconds ago) due to excessive rate
2020-11-17T14:04:59Z|00709|lflow|WARN|error parsing match "((ct.new && !ct.est) || (!ct.new && ct.est && !ct.rpl && ct_label.blocked == 1)) && (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4)": Syntax error at `$pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4' expecting address set name.
What that is happening:
Only security groups that are not in use by other ports can be deleted. If a security group referenced as remote by other group is removed, the security group rule is deleted in DB because of the references in DB tables. It means there is no code triggered in the Neutron code. The corresponding port group is deleted in OVN but OVN doesn't have any on delete cascade functionality like SQL has. It means ACL remains in the OVN DB using port group that no longer exists. It has no impact on the traffic because the deleted security group was not in use. |
Neutron does not clean up ACLs in OVN NB DB when a remote security group is deleted in Neutron. This causes some warning in OVN logs. This issue does not impact functionality but it would be great if we could clean this up and make those logs go away. I think this BZ can be marked as low priority and low hanging fruit.
Reproducing steps:
1. A security group rule which has a rule that uses a remote security group
openstack security group rule list 44038bce-38fd-4219-ae22-bafaad9bbde9
+--------------------------------------+-------------+----------+------------+--------------------------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+----------+------------+--------------------------------------+
| 1155ffc7-9a99-41c7-bd65-fac33b719973 | icmp | None | | 4b0f872b-22ae-4ac2-b54e-ea1678a88dd5 |
| 8e4cdd63-c3e8-411f-8d72-c5cced5d4fc8 | None | None | | None |
| b72b29df-d5f1-4815-92ae-71b1cdc35f97 | None | None | | None |
+--------------------------------------+-------------+----------+------------+--------------------------------------+
Here is a view in ovn nb db
docker exec -it ovn-dbs-bundle-docker-1 ovn-nbctl acl-list pg_44038bce_38fd_4219_ae22_bafaad9bbde9
from-lport 1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4) allow-related
from-lport 1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip6) allow-related
to-lport 1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related
2. now remove the remote security group and check the rules in the first security group
openstack security group delete 4b0f872b-22ae-4ac2-b54e-ea1678a88dd5
openstack security group rule list 44038bce-38fd-4219-ae22-bafaad9bbde9
+--------------------------------------+-------------+----------+------------+-----------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+----------+------------+-----------------------+
| 8e4cdd63-c3e8-411f-8d72-c5cced5d4fc8 | None | None | | None |
| b72b29df-d5f1-4815-92ae-71b1cdc35f97 | None | None | | None |
+--------------------------------------+-------------+----------+------------+-----------------------+
So from neutron the security group rule is removed
3. check the acl in ovn again and see they are still there
docker exec -it ovn-dbs-bundle-docker-1 ovn-nbctl acl-list pg_44038bce_38fd_4219_ae22_bafaad9bbde9
from-lport 1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4) allow-related
from-lport 1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip6) allow-related
to-lport 1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related
this causes warnings to be generated in ovn
2020-11-17T14:04:59Z|00708|lflow|WARN|Dropped 1 log messages in last 917 seconds (most recently, 917 seconds ago) due to excessive rate
2020-11-17T14:04:59Z|00709|lflow|WARN|error parsing match "((ct.new && !ct.est) || (!ct.new && ct.est && !ct.rpl && ct_label.blocked == 1)) && (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4)": Syntax error at `$pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4' expecting address set name.
What that is happening:
Only security groups that are not in use by other ports can be deleted. If a security group referenced as remote by other group is removed, the security group rule is deleted in DB because of the references in DB tables. It means there is no code triggered in the Neutron code. The corresponding port group is deleted in OVN but OVN doesn't have any on delete cascade functionality like SQL has. It means ACL remains in the OVN DB using port group that no longer exists. It has no impact on the traffic because the deleted security group was not in use. |
|
| 2022-08-05 16:43:10 |
Elvira García Ruiz |
neutron: importance |
Undecided |
Low |
|
| 2022-08-05 16:43:16 |
Elvira García Ruiz |
neutron: status |
New |
Confirmed |
|
| 2022-08-09 15:57:10 |
Rodolfo Alonso |
neutron: assignee |
|
Rodolfo Alonso (rodolfo-alonso-hernandez) |
|
| 2022-08-10 13:49:50 |
OpenStack Infra |
neutron: status |
Confirmed |
In Progress |
|
| 2022-08-24 14:40:55 |
Rodolfo Alonso |
tags |
|
backport-potential |
|
| 2022-08-24 17:32:43 |
OpenStack Infra |
neutron: status |
In Progress |
Fix Released |
|
| 2022-08-26 09:55:00 |
OpenStack Infra |
tags |
backport-potential |
backport-potential in-stable-xena |
|
| 2022-08-29 13:49:42 |
OpenStack Infra |
tags |
backport-potential in-stable-xena |
backport-potential in-stable-wallaby in-stable-xena |
|
| 2022-08-30 21:16:31 |
OpenStack Infra |
tags |
backport-potential in-stable-wallaby in-stable-xena |
backport-potential in-stable-wallaby in-stable-xena in-stable-yoga |
|
