master : C8 ovb jobs failing with Could not load 'oslo_cache.etcd3gw': No module named 'etcd3gw''

[Pooja Jadhav]

All CentOS-8 ovb jobs are failing with below tracback :

021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server [req-9b8bf1e9-a7ed-4b2c-964e-9414f7757317 - - - - -] Exception during message handling: ironic.common.exception.InvalidStateRequested: Cannot continue deploying on da880092-3289-4f00-be96-ae4769f2a0af. Node is in active state; should be in one of wait call-back, deploying
2021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/ironic_lib/metrics.py", line 59, in wrapped
2021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/ironic/conductor/manager.py", line 882, in continue_node_deploy
2021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server 'deploy_state': ', '.join(expected_states)})
2021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server ironic.common.exception.InvalidStateRequested: Cannot continue deploying on da880092-3289-4f00-be96-ae4769f2a0af. Node is in active state; should be in one of wait call-back, deploying
2021-05-10 03:19:40.827 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR oslo_messaging.rpc.server
2021-05-10 03:07:16.713 ERROR /var/log/containers/neutron/server.log: 30 ERROR stevedore.extension [req-815c46f6-3ad8-4e2c-ac36-09847faa594e f1569de1c9d6477a87a54c085ec3e7fe 99125963304f44ff8c002e8d66476833 - default default] Could not load 'oslo_cache.etcd3gw': No module named 'etcd3gw': ModuleNotFoundError: No module named 'etcd3gw'
2021-05-10 03:05:26.895 ERROR /var/log/containers/neutron/openvswitch-agent.log: 104159 ERROR ovsdbapp.backend.ovs_idl.idlutils [-] Unable to open stream to tcp:127.0.0.1:6640 to retrieve schema: Connection refused

Affacted master periodic jobs :

periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master
periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master
periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master
periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master
periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master

References :

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/b70a078/logs/undercloud/var/log/extra/errors.txt.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/3a1e691/logs/undercloud/var/log/extra/errors.txt.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/fffcad7/logs/undercloud/var/log/extra/errors.txt.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/888de2e/logs/undercloud/var/log/extra/errors.txt.txt.gz

[Ronelle Landy (rlandy)]

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/68ad336/logs/undercloud/var/log/extra/errors.txt.txt.gz

ModuleNotFoundError: No module named 'etcd3gw' shows up in passing jobs.

The issue here seems to be in:

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/888de2e/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz