OVN security group logging burst limit has an unexpected value for stateless security groups

[Elvira García Ruiz]

If we set the meter band burst limit for a certain number, we expect to have a a total number of logs almost equal (~10% difference) to the rate_limit + burst_limit for 1 second timeframe. This is true for stateful security groups, but not for stateless security groups. The result of tuning the burst limit should be equal for both stateless and stateful. Currently, stateless security groups outputs exactly double of the expected logs.

I have already discussed this with OVN folks and it looks like it is not on them how stateless and stateful connections work, it's kernel-wise. Therefore, the most immediate way of fixing this is on Neutron itself.

* Step-by-step reproduction steps using Devstack:

See current number of log in ovn-controller.log
C1=$(sudo grep acl_log /opt/stack/logs/ovn-controller.log | tail -n1 | cut -d "|" -f 2); echo $C1

Send big amount of ICMP requests from the undercloud node for less than a second:
sudo ping 172.24.4.223 -i 0.002 -c 500 | tail -n4 #stateless

sudo ping 172.24.4.129 -i 0.002 -c 500 | tail -n4 #stateful
o traffic loss should be observed
Check the last log entry ID and so calculate the amount of logs generated from the last time:
C2=$(sudo grep acl_log /opt/stack/logs/ovn-controller.log | tail -n1 | cut -d "|" -f 2); echo $C2
Log amount (C2 - C1) should be almost equal (~10% difference) to the rate_limit + burst_limit.
Result in between 112 and 138, according to default limit values.

* Results on my environment
burst limit: 25 rate limit: 100

RESULTS (C2-C1):

For stateless security groups: 248
For stateful security groups: 124

Expected result: Aproximately 125 for both

More information at: https://bugzilla.redhat.com/show_bug.cgi?id=2212952

[Mamatisa Nurmatov (isabek)]

Patch: https://review.opendev.org/c/openstack/neutron/+/892648

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/893460

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/892648
Committed: https://opendev.org/openstack/neutron/commit/a3a113aedbf40fdb8d90179033773a9d225a0735
Submitter: "Zuul (22348)"
Branch: master

commit a3a113aedbf40fdb8d90179033773a9d225a0735
Author: Elvira García <email address hidden>
Date: Thu Aug 24 10:31:30 2023 +0200

    [OVN] Fix rate and burst for stateless security groups

    Right now, as per kernel limitation, the burst limit is not correctly
    enforcing the rate and burst when using the ovn "log-related" option and
    stateless security groups. We log exactly double the burst. Creating a
    new meter that limits the rate and burst to half of the expected ones is
    a workaround that solves the issue.

    Closes-bug: #2032929

    Signed-off-by: Elvira García <email address hidden>
    Change-Id: Ib0047d38c58bcebb23c8887e7934987ff8c8a432

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/894846

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 23.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/895465

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/895663

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/895783

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/895785

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/895465
Committed: https://opendev.org/openstack/neutron/commit/67bd591c5b9280f1af7c8a4942dcbb3bd1270cfb
Submitter: "Zuul (22348)"
Branch: master

commit 67bd591c5b9280f1af7c8a4942dcbb3bd1270cfb
Author: Elvira García <email address hidden>
Date: Fri Sep 15 16:33:43 2023 +0200

    Use safer methods to get security groups on security group logging

    There is a chance on real environment that a port group doesn't have any
    correspondent security group (and there are maintenance tasks that will
    remove them). This patch avoids a DriverError from Neutron in case we
    are in an environment with a port group that was mistakenly left over
    due to any reason. Instead, a Warning log will be raised.

    Related-bug: #2032929
    Change-Id: I42208557c8522d6fbc29df8a3c7d0367cace31e4

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/896335

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/896335
Committed: https://opendev.org/openstack/neutron/commit/f1638bb6d130b5150bcdae669d2ce43bd6b9d688
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit f1638bb6d130b5150bcdae669d2ce43bd6b9d688
Author: Elvira García <email address hidden>
Date: Fri Sep 15 16:33:43 2023 +0200

    Use safer methods to get security groups on security group logging

    There is a chance on real environment that a port group doesn't have any
    correspondent security group (and there are maintenance tasks that will
    remove them). This patch avoids a DriverError from Neutron in case we
    are in an environment with a port group that was mistakenly left over
    due to any reason. Instead, a Warning log will be raised.

    Related-bug: #2032929
    Change-Id: I42208557c8522d6fbc29df8a3c7d0367cace31e4
    (cherry picked from commit 67bd591c5b9280f1af7c8a4942dcbb3bd1270cfb)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/896614

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elvira García Ruiz <email address hidden>" on branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/896614
Reason: I did not want to create a different commit for this, I wanted to modify 893460

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elvira García Ruiz <email address hidden>" on branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/894846
Reason: outdate

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elvira García Ruiz <email address hidden>" on branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/895663
Reason: outdated

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elvira García Ruiz <email address hidden>" on branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/895783
Reason: outdated

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elvira García Ruiz <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/895785
Reason: outdated

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/897470

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/897530

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/897531

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/897532

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/893460
Committed: https://opendev.org/openstack/neutron/commit/226220ac3c9f651ea4d5b564f7cbe1acd3a272a8
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 226220ac3c9f651ea4d5b564f7cbe1acd3a272a8
Author: Elvira García <email address hidden>
Date: Thu Aug 24 10:31:30 2023 +0200

    [OVN] Fix rate and burst for stateless security groups

    Right now, as per kernel limitation, the burst limit is not correctly
    enforcing the rate and burst when using the ovn "log-related" option and
    stateless security groups. We log exactly double the burst. Creating a
    new meter that limits the rate and burst to half of the expected ones is
    a workaround that solves the issue.

    Closes-bug: #2032929

    Conflicts:
     - neutron/services/logapi/drivers/ovn/driver.py
     - neutron/tests/functional/services/logapi/drivers/ovn/test_driver.py

    Signed-off-by: Elvira García <email address hidden>
    Change-Id: Ib0047d38c58bcebb23c8887e7934987ff8c8a432
    (cherry picked from commit a3a113aedbf40fdb8d90179033773a9d225a0735)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/896614
Committed: https://opendev.org/openstack/neutron/commit/7066974509bb0ec9c6d389feea4c3e1600ab4da2
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 7066974509bb0ec9c6d389feea4c3e1600ab4da2
Author: Elvira García <email address hidden>
Date: Fri Sep 15 16:33:43 2023 +0200

    Use safer methods to get security groups on security group logging

    There is a chance on real environment that a port group doesn't have any
    correspondent security group (and there are maintenance tasks that will
    remove them). This patch avoids a DriverError from Neutron in case we
    are in an environment with a port group that was mistakenly left over
    due to any reason. Instead, a Warning log will be raised.

    Related-bug: #2032929
    Change-Id: I42208557c8522d6fbc29df8a3c7d0367cace31e4
    (cherry picked from commit 67bd591c5b9280f1af7c8a4942dcbb3bd1270cfb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/894846
Committed: https://opendev.org/openstack/neutron/commit/3152bc14f86d4616b363cb7b31648c534d8eb959
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 3152bc14f86d4616b363cb7b31648c534d8eb959
Author: Elvira García <email address hidden>
Date: Thu Aug 24 10:31:30 2023 +0200

    [OVN] Fix rate and burst for stateless security groups

    Right now, as per kernel limitation, the burst limit is not correctly
    enforcing the rate and burst when using the ovn "log-related" option and
    stateless security groups. We log exactly double the burst. Creating a
    new meter that limits the rate and burst to half of the expected ones is
    a workaround that solves the issue.

    Closes-bug: #2032929

    Conflicts:
     - neutron/services/logapi/drivers/ovn/driver.py
     - neutron/tests/functional/services/logapi/drivers/ovn/test_driver.py

    Signed-off-by: Elvira García <email address hidden>
    Change-Id: Ib0047d38c58bcebb23c8887e7934987ff8c8a432
    (cherry picked from commit a3a113aedbf40fdb8d90179033773a9d225a0735)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/895783
Committed: https://opendev.org/openstack/neutron/commit/6172f4233ef0bd1199deacefbf81f3104b5ccc7a
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 6172f4233ef0bd1199deacefbf81f3104b5ccc7a
Author: Elvira García <email address hidden>
Date: Thu Aug 24 10:31:30 2023 +0200

    [OVN] Fix rate and burst for stateless security groups

    Right now, as per kernel limitation, the burst limit is not correctly
    enforcing the rate and burst when using the ovn "log-related" option and
    stateless security groups. We log exactly double the burst. Creating a
    new meter that limits the rate and burst to half of the expected ones is
    a workaround that solves the issue.

    Closes-bug: #2032929

    Conflicts:
     - neutron/services/logapi/drivers/ovn/driver.py
     - neutron/tests/functional/services/logapi/drivers/ovn/test_driver.py

    Signed-off-by: Elvira García <email address hidden>
    Change-Id: Ib0047d38c58bcebb23c8887e7934987ff8c8a432
    (cherry picked from commit a3a113aedbf40fdb8d90179033773a9d225a0735)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/897470
Committed: https://opendev.org/openstack/neutron/commit/373d8ee969e7b387960952919794d9724ce2bcfc
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 373d8ee969e7b387960952919794d9724ce2bcfc
Author: Elvira García <email address hidden>
Date: Fri Sep 15 16:33:43 2023 +0200

    Use safer methods to get security groups on security group logging

    There is a chance on real environment that a port group doesn't have any
    correspondent security group (and there are maintenance tasks that will
    remove them). This patch avoids a DriverError from Neutron in case we
    are in an environment with a port group that was mistakenly left over
    due to any reason. Instead, a Warning log will be raised.

    Related-bug: #2032929
    Change-Id: I42208557c8522d6fbc29df8a3c7d0367cace31e4
    (cherry picked from commit 67bd591c5b9280f1af7c8a4942dcbb3bd1270cfb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/897531
Committed: https://opendev.org/openstack/neutron/commit/53484b897b76d3dd8a9f8c1992591dfb52f1dece
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 53484b897b76d3dd8a9f8c1992591dfb52f1dece
Author: Elvira García <email address hidden>
Date: Fri Sep 15 16:33:43 2023 +0200

    Use safer methods to get security groups on security group logging

    There is a chance on real environment that a port group doesn't have any
    correspondent security group (and there are maintenance tasks that will
    remove them). This patch avoids a DriverError from Neutron in case we
    are in an environment with a port group that was mistakenly left over
    due to any reason. Instead, a Warning log will be raised.

    Related-bug: #2032929
    Change-Id: I42208557c8522d6fbc29df8a3c7d0367cace31e4
    (cherry picked from commit 67bd591c5b9280f1af7c8a4942dcbb3bd1270cfb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/895663
Committed: https://opendev.org/openstack/neutron/commit/e82afb138de80e6213831ac01ffcf9f2318c77f7
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit e82afb138de80e6213831ac01ffcf9f2318c77f7
Author: Elvira García <email address hidden>
Date: Thu Aug 24 10:31:30 2023 +0200

    [OVN] Fix rate and burst for stateless security groups

    Right now, as per kernel limitation, the burst limit is not correctly
    enforcing the rate and burst when using the ovn "log-related" option and
    stateless security groups. We log exactly double the burst. Creating a
    new meter that limits the rate and burst to half of the expected ones is
    a workaround that solves the issue.

    Closes-bug: #2032929

    Conflicts:
     - neutron/services/logapi/drivers/ovn/driver.py
     - neutron/tests/functional/services/logapi/drivers/ovn/test_driver.py

    Signed-off-by: Elvira García <email address hidden>
    Change-Id: Ib0047d38c58bcebb23c8887e7934987ff8c8a432
    (cherry picked from commit a3a113aedbf40fdb8d90179033773a9d225a0735)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/897530
Committed: https://opendev.org/openstack/neutron/commit/bb7d13930d0d13eb0238e3c53409176994e2b97f
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit bb7d13930d0d13eb0238e3c53409176994e2b97f
Author: Elvira García <email address hidden>
Date: Fri Sep 15 16:33:43 2023 +0200

    Use safer methods to get security groups on security group logging

    There is a chance on real environment that a port group doesn't have any
    correspondent security group (and there are maintenance tasks that will
    remove them). This patch avoids a DriverError from Neutron in case we
    are in an environment with a port group that was mistakenly left over
    due to any reason. Instead, a Warning log will be raised.

    Related-bug: #2032929
    Change-Id: I42208557c8522d6fbc29df8a3c7d0367cace31e4
    (cherry picked from commit 67bd591c5b9280f1af7c8a4942dcbb3bd1270cfb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/895785
Committed: https://opendev.org/openstack/neutron/commit/c4acdc084a5ffca9a904cf8308d92a6c99bdffbc
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit c4acdc084a5ffca9a904cf8308d92a6c99bdffbc
Author: Elvira García <email address hidden>
Date: Thu Aug 24 10:31:30 2023 +0200

    [OVN] Fix rate and burst for stateless security groups

    Right now, as per kernel limitation, the burst limit is not correctly
    enforcing the rate and burst when using the ovn "log-related" option and
    stateless security groups. We log exactly double the burst. Creating a
    new meter that limits the rate and burst to half of the expected ones is
    a workaround that solves the issue.

    Closes-bug: #2032929

    Conflicts:
     - neutron/services/logapi/drivers/ovn/driver.py
     - neutron/tests/functional/services/logapi/drivers/ovn/test_driver.py

    Signed-off-by: Elvira García <email address hidden>
    Change-Id: Ib0047d38c58bcebb23c8887e7934987ff8c8a432
    (cherry picked from commit a3a113aedbf40fdb8d90179033773a9d225a0735)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/897532
Committed: https://opendev.org/openstack/neutron/commit/1bfc75a71b6f2fb995dfc7fc0365dec5410d7e31
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 1bfc75a71b6f2fb995dfc7fc0365dec5410d7e31
Author: Elvira García <email address hidden>
Date: Fri Sep 15 16:33:43 2023 +0200

    Use safer methods to get security groups on security group logging

    There is a chance on real environment that a port group doesn't have any
    correspondent security group (and there are maintenance tasks that will
    remove them). This patch avoids a DriverError from Neutron in case we
    are in an environment with a port group that was mistakenly left over
    due to any reason. Instead, a Warning log will be raised.

    Related-bug: #2032929
    Change-Id: I42208557c8522d6fbc29df8a3c7d0367cace31e4
    (cherry picked from commit 67bd591c5b9280f1af7c8a4942dcbb3bd1270cfb)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron xena-eom release.