[OVN] Missing lock check in check_for_mcast_flood_reports

[Lucas Alvares Gomes]

The maintenance method check_for_mcast_flood_reports() is not checking if that instance of the maintenance task has a proper lock before trying to execute a transaction. If more than one controller is deployed only 1 instance will hold a valid lock the rest will fail with:

2021-05-02 14:23:29.193 599 ERROR futurist.periodics [-] Failed to call periodic 'networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_mcast_flood_reports' (it runs every 600.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2021-05-02 14:23:29.193 599 ERROR futurist.periodics Traceback (most recent call last):
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
2021-05-02 14:23:29.193 599 ERROR futurist.periodics yield self._nested_txns_map[cur_thread_id]
2021-05-02 14:23:29.193 599 ERROR futurist.periodics KeyError: 139844066981736
2021-05-02 14:23:29.193 599 ERROR futurist.periodics
2021-05-02 14:23:29.193 599 ERROR futurist.periodics During handling of the above exception, another exception occurred:
2021-05-02 14:23:29.193 599 ERROR futurist.periodics
2021-05-02 14:23:29.193 599 ERROR futurist.periodics Traceback (most recent call last):
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 290, in run
2021-05-02 14:23:29.193 599 ERROR futurist.periodics work()
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 64, in __call__
2021-05-02 14:23:29.193 599 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 178, in decorator
2021-05-02 14:23:29.193 599 ERROR futurist.periodics return f(*args, **kwargs)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 667, in check_for_mcast_flood_reports
2021-05-02 14:23:29.193 599 ERROR futurist.periodics for port in self._nb_idl.lsp_list().execute(check_error=True):
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 42, in execute
2021-05-02 14:23:29.193 599 ERROR futurist.periodics t.add(self)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2021-05-02 14:23:29.193 599 ERROR futurist.periodics next(self.gen)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 196, in transaction
2021-05-02 14:23:29.193 599 ERROR futurist.periodics yield t
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2021-05-02 14:23:29.193 599 ERROR futurist.periodics next(self.gen)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2021-05-02 14:23:29.193 599 ERROR futurist.periodics del self._nested_txns_map[cur_thread_id]
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2021-05-02 14:23:29.193 599 ERROR futurist.periodics self.result = self.commit()
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2021-05-02 14:23:29.193 599 ERROR futurist.periodics raise result.ex
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run
2021-05-02 14:23:29.193 599 ERROR futurist.periodics txn.results.put(txn.do_commit())
2021-05-02 14:23:29.193 599 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
2021-05-02 14:23:29.193 599 ERROR futurist.periodics raise RuntimeError(msg)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2021-05-02 14:23:29.193 599 ERROR futurist.periodics

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/789547

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/789579

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/789580

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/789581

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789547
Committed: https://opendev.org/openstack/neutron/commit/317407fe61b76b8f0c250ce6fb18d93dd0b667d0
Submitter: "Zuul (22348)"
Branch: master

commit 317407fe61b76b8f0c250ce6fb18d93dd0b667d0
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue May 4 11:33:20 2021 +0100

    [OVN] Check for lock in check_for_mcast_flood_reports

    The maintenance method check_for_mcast_flood_reports() was not checking
    if that instance of the maintenance task has a valid lock before trying
    to execute a transaction.

    This patch adds such check so instances of the maintenance task without
    a valid lock will just return.

    Change-Id: I791026058b37f06d1a19b5fb6823b9dbb2a3eca5
    Closes-Bug: #1927077
    Signed-off-by: Lucas Alvares Gomes <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789580
Committed: https://opendev.org/openstack/neutron/commit/b2fd99922032c82ec6a7e25bf4236327347f377c
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit b2fd99922032c82ec6a7e25bf4236327347f377c
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue May 4 11:33:20 2021 +0100

    [OVN] Check for lock in check_for_mcast_flood_reports

    The maintenance method check_for_mcast_flood_reports() was not checking
    if that instance of the maintenance task has a valid lock before trying
    to execute a transaction.

    This patch adds such check so instances of the maintenance task without
    a valid lock will just return.

    Change-Id: I791026058b37f06d1a19b5fb6823b9dbb2a3eca5
    Closes-Bug: #1927077
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 317407fe61b76b8f0c250ce6fb18d93dd0b667d0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789581
Committed: https://opendev.org/openstack/neutron/commit/2e49c2b743e055a3e952d32d4f89012a629dfba2
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 2e49c2b743e055a3e952d32d4f89012a629dfba2
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue May 4 11:33:20 2021 +0100

    [OVN] Check for lock in check_for_mcast_flood_reports

    The maintenance method check_for_mcast_flood_reports() was not checking
    if that instance of the maintenance task has a valid lock before trying
    to execute a transaction.

    This patch adds such check so instances of the maintenance task without
    a valid lock will just return.

    Change-Id: I791026058b37f06d1a19b5fb6823b9dbb2a3eca5
    Closes-Bug: #1927077
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 317407fe61b76b8f0c250ce6fb18d93dd0b667d0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789579
Committed: https://opendev.org/openstack/neutron/commit/fc00c0959b8b5bb9e5c860be14cfb10e2c80f348
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit fc00c0959b8b5bb9e5c860be14cfb10e2c80f348
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue May 4 11:33:20 2021 +0100

    [OVN] Check for lock in check_for_mcast_flood_reports

    The maintenance method check_for_mcast_flood_reports() was not checking
    if that instance of the maintenance task has a valid lock before trying
    to execute a transaction.

    This patch adds such check so instances of the maintenance task without
    a valid lock will just return.

    Change-Id: I791026058b37f06d1a19b5fb6823b9dbb2a3eca5
    Closes-Bug: #1927077
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 317407fe61b76b8f0c250ce6fb18d93dd0b667d0)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.3.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.1.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn train-eol release.