tripleo

ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router with name=neutron-<uuid> is failing on featureset001 and featureset035 master

Bug #1908976 reported by Bhagyashri Shewale on 2020-12-22

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	High	Unassigned	tripleo wallaby-2

Bug Description

Tempest test are failing [1] because of this issue and that are causing to fail the periodic jobs on master (featureset001 and featureset035)

Error logs:

2020-12-22 03:22:16.877 ERROR /var/log/containers/glance/api.log: 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command [req-960b0439-2546-4e0f-9450-82d1b036906d bbae629fc2ab4f0e9cdd44ca300a9a11 84e551a35077458d922ed47a94c4bf56 - default default] Error executing command: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router with name=neutron-2df67978-cf94-4506-83a3-b7c5c5a98477
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 39, in execute
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command self.run_idl(None)
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 328, in run_idl
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command self.result = self.api.lookup(self.table, self.record)
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 177, in lookup
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command return self._lookup(table, record)
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 224, in _lookup
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command row = idlutils.row_by_value(self, rl.table, rl.column, record)
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command raise RowNotFound(table=table, col=column, match=match)
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router with name=neutron-2df67978-cf94-4506-83a3-b7c5c5a98477
2020-12-22 04:01:44.736 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command
2020-12-22 04:01:46.618 ERROR /var/log/containers/neutron/server.log: 28 ERROR ovsdbapp.backend.ovs_idl.command [req-960b0439-2546-4e0f-9450-82d1b036906d bbae629fc2ab4f0e9cdd44ca300a9a11 84e551a35077458d922ed47a94c4bf56 - default default] Error executing command: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router with name=neutron-2df67978-cf94-4506-83a3-b7c5c5a98477

Affected jobs:
1. periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master
2. periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master

[1]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/959c026/logs/undercloud/var/log/tempest/stestr_results.html.gz

[2]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/6146346/logs/undercloud/var/log/tempest/stestr_results.html.gz

[3]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/959c026/logs/overcloud-controller-0/var/log/extra/errors.txt.txt.gz

[4]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/6146346/logs/overcloud-controller-0/var/log/extra/errors.txt.txt.gz

[5]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/6146346/logs/overcloud-controller-0/var/log/containers/neutron/server.log.gz

Tags:

Bhagyashri Shewale (bhagyashri-shewale) on 2020-12-22

Changed in tripleo:
milestone:	none → wallaby-2

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-12-22:

The error can be found in

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/6146346/logs/overcloud-controller-1/var/log/containers/neutron/server.log.gz

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-12-22:

I suspect perhaps the a diff in libvirt packages *may* be the issue here..
https://termbin.com/zr6y

Revision history for this message

Bhagyashri Shewale (bhagyashri-shewale) wrote on 2020-12-23:

Hi All,

Here is the related fix: [1]
By testing with the help of testproject patch [2] using of Depends on above patch [1] it's still failing with same issue.

[1]: https://review.opendev.org/c/openstack/neutron/+/753847 (Avoid race condition when processing RowEvents)

[2]: https://review.rdoproject.org/r/31417 [DNM] Test bug/1908976

Revision history for this message

Sagi (Sergey) Shnaidman (sshnaidm) wrote on 2020-12-28:

I see everywhere in tests error "nova.exception.CinderConnectionFailed"

In nova logs there are a lot of SSL errors:

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
raise MaxRetryError(_pool, url, error or ResponseError(cause))
2020-12-28 15:02:30.280 17 ERROR nova.api.openstack.wsgi urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.0.0.5', port=13776): Max retries exceeded with url: /v3/a67134b437ab4ba3b11aeb8f81d9ed94/volumes/fc687f24-f221-44c4-b4ff-c1f5f2374cc7 (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))