[OVN] Neutron server floods logs with hash ring messages on startup

[Lucas Alvares Gomes]

Reported at: https://bugzilla.redhat.com/show_bug.cgi?id=2125828

Neutron server issues over 300 messages per second during startup with debug on:

2022-09-10 21:55:32.998 55 DEBUG networking_ovn.common.hash_ring_manager [-] Disallow caching, nodes 0<26 _wait_startup_before_caching /usr/lib/python3.6/site-packages/networking_ovn/common/hash_ring_manager.py:61

This message is logged when the number of nodes in the hash ring is different than the number of API workers at Neutron's startup. The hash ring waits until all API workers are connected to OVSDB prior to building the hash ring cache.

We need to rate limit this message, in case there are problems with the API workers not being able to connected to OVSDB this message does not spam the Neutron logs.

The message itself is still useful IMHO, for knowing that the hash ring cache is not yet built.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/857463

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/857463
Committed: https://opendev.org/openstack/neutron/commit/9655466763282f47a06862a47f7f31b48130277e
Submitter: "Zuul (22348)"
Branch: master

commit 9655466763282f47a06862a47f7f31b48130277e
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Sep 13 16:11:37 2022 +0100

    [OVN] Rate limit the "Disallow caching" log from hash ring

    This patch adds a condition prior to logging the "Disallow caching" log
    message from the hash ring.

    Prior to this patch, this message was logged when the number of nodes
    connected to the hash ring was different from the number of API workers
    at Neutron's startup. This is because the hash ring waits until all API
    workers are connected to build the hash ring cache.

    With this patch, we will only log the message once (per worker) until
    the number of connected nodes changes. When nodes connect and the cache
    is built the _wait_startup_before_caching() is no longer used until the
    service is restarted again.

    Change-Id: I4f62b723083215483a2277cfcb798506671e1a2d
    Closes-Bug: 1989480
    Signed-off-by: Lucas Alvares Gomes <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/858728

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/858729

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/858730

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/858731

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/858732

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/858733

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858731
Committed: https://opendev.org/openstack/neutron/commit/4b6a8a2157fd3aa93bf2681686ded94df1d6da62
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 4b6a8a2157fd3aa93bf2681686ded94df1d6da62
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Sep 13 16:11:37 2022 +0100

    [OVN] Rate limit the "Disallow caching" log from hash ring

    This patch adds a condition prior to logging the "Disallow caching" log
    message from the hash ring.

    Prior to this patch, this message was logged when the number of nodes
    connected to the hash ring was different from the number of API workers
    at Neutron's startup. This is because the hash ring waits until all API
    workers are connected to build the hash ring cache.

    With this patch, we will only log the message once (per worker) until
    the number of connected nodes changes. When nodes connect and the cache
    is built the _wait_startup_before_caching() is no longer used until the
    service is restarted again.

    Change-Id: I4f62b723083215483a2277cfcb798506671e1a2d
    Closes-Bug: 1989480
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 9655466763282f47a06862a47f7f31b48130277e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858732
Committed: https://opendev.org/openstack/neutron/commit/b3b2c554fa85c6d58e641e8e1cfd3c1ca4e9a4cb
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit b3b2c554fa85c6d58e641e8e1cfd3c1ca4e9a4cb
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Sep 13 16:11:37 2022 +0100

    [OVN] Rate limit the "Disallow caching" log from hash ring

    This patch adds a condition prior to logging the "Disallow caching" log
    message from the hash ring.

    Prior to this patch, this message was logged when the number of nodes
    connected to the hash ring was different from the number of API workers
    at Neutron's startup. This is because the hash ring waits until all API
    workers are connected to build the hash ring cache.

    With this patch, we will only log the message once (per worker) until
    the number of connected nodes changes. When nodes connect and the cache
    is built the _wait_startup_before_caching() is no longer used until the
    service is restarted again.

    Change-Id: I4f62b723083215483a2277cfcb798506671e1a2d
    Closes-Bug: 1989480
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 9655466763282f47a06862a47f7f31b48130277e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858728
Committed: https://opendev.org/openstack/neutron/commit/9f36ab53b49c1f83e6df7737c4e92b41e9658660
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 9f36ab53b49c1f83e6df7737c4e92b41e9658660
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Sep 13 16:11:37 2022 +0100

    [OVN] Rate limit the "Disallow caching" log from hash ring

    This patch adds a condition prior to logging the "Disallow caching" log
    message from the hash ring.

    Prior to this patch, this message was logged when the number of nodes
    connected to the hash ring was different from the number of API workers
    at Neutron's startup. This is because the hash ring waits until all API
    workers are connected to build the hash ring cache.

    With this patch, we will only log the message once (per worker) until
    the number of connected nodes changes. When nodes connect and the cache
    is built the _wait_startup_before_caching() is no longer used until the
    service is restarted again.

    Change-Id: I4f62b723083215483a2277cfcb798506671e1a2d
    Closes-Bug: 1989480
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 9655466763282f47a06862a47f7f31b48130277e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858730
Committed: https://opendev.org/openstack/neutron/commit/564e83635885639177835e1cfcd3c73e66d82723
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 564e83635885639177835e1cfcd3c73e66d82723
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Sep 13 16:11:37 2022 +0100

    [OVN] Rate limit the "Disallow caching" log from hash ring

    This patch adds a condition prior to logging the "Disallow caching" log
    message from the hash ring.

    Prior to this patch, this message was logged when the number of nodes
    connected to the hash ring was different from the number of API workers
    at Neutron's startup. This is because the hash ring waits until all API
    workers are connected to build the hash ring cache.

    With this patch, we will only log the message once (per worker) until
    the number of connected nodes changes. When nodes connect and the cache
    is built the _wait_startup_before_caching() is no longer used until the
    service is restarted again.

    Change-Id: I4f62b723083215483a2277cfcb798506671e1a2d
    Closes-Bug: 1989480
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 9655466763282f47a06862a47f7f31b48130277e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858729
Committed: https://opendev.org/openstack/neutron/commit/7721c6982b2cddee02d807cf4905a54f90358cd9
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 7721c6982b2cddee02d807cf4905a54f90358cd9
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Sep 13 16:11:37 2022 +0100

    [OVN] Rate limit the "Disallow caching" log from hash ring

    This patch adds a condition prior to logging the "Disallow caching" log
    message from the hash ring.

    Prior to this patch, this message was logged when the number of nodes
    connected to the hash ring was different from the number of API workers
    at Neutron's startup. This is because the hash ring waits until all API
    workers are connected to build the hash ring cache.

    With this patch, we will only log the message once (per worker) until
    the number of connected nodes changes. When nodes connect and the cache
    is built the _wait_startup_before_caching() is no longer used until the
    service is restarted again.

    Change-Id: I4f62b723083215483a2277cfcb798506671e1a2d
    Closes-Bug: 1989480
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 9655466763282f47a06862a47f7f31b48130277e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858733
Committed: https://opendev.org/openstack/neutron/commit/d0ade7e48c0f3da88e8e90aa1ff54dbd3d8efc12
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit d0ade7e48c0f3da88e8e90aa1ff54dbd3d8efc12
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Sep 13 16:11:37 2022 +0100

    [OVN] Rate limit the "Disallow caching" log from hash ring

    This patch adds a condition prior to logging the "Disallow caching" log
    message from the hash ring.

    Prior to this patch, this message was logged when the number of nodes
    connected to the hash ring was different from the number of API workers
    at Neutron's startup. This is because the hash ring waits until all API
    workers are connected to build the hash ring cache.

    With this patch, we will only log the message once (per worker) until
    the number of connected nodes changes. When nodes connect and the cache
    is built the _wait_startup_before_caching() is no longer used until the
    service is restarted again.

    Change-Id: I4f62b723083215483a2277cfcb798506671e1a2d
    Closes-Bug: 1989480
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 9655466763282f47a06862a47f7f31b48130277e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron victoria-eom release.