http://logs.openstack.org/96/644596/1/check/grenade-py3/c5059ab/logs/screen-c-bak.txt.gz
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service [-] Error starting thread.: ModuleNotFoundError: No module named 'etcd3gw'
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service Traceback (most recent call last):
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/oslo_service/service.py", line 796, in run_service
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service service.start()
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service File "/opt/stack/old/cinder/cinder/service.py", line 219, in start
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service coordination.COORDINATOR.start()
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service File "/opt/stack/old/cinder/cinder/coordination.py", line 66, in start
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service cfg.CONF.coordination.backend_url, member_id)
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/tooz/coordination.py", line 800, in get_coordinator
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service invoke_args=(member_id, parsed_url, options)).driver
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/stevedore/driver.py", line 61, in __init__
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service warn_on_missing_entrypoint=warn_on_missing_entrypoint
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/stevedore/named.py", line 81, in __init__
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service verify_requirements)
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 203, in _load_plugins
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service self._on_load_failure_callback(self, ep, err)
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 195, in _load_plugins
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service verify_requirements,
Mar 19 16:14:03.940298 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/stevedore/named.py", line 158, in _load_one_plugin
Mar 19 16:14:03.943132 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service verify_requirements,
Mar 19 16:14:03.943132 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 223, in _load_one_plugin
Mar 19 16:14:03.943132 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service plugin = ep.resolve()
Mar 19 16:14:03.943132 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 2417, in resolve
Mar 19 16:14:03.943132 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service module = __import__(self.module_name, fromlist=['__name__'], level=0)
Mar 19 16:14:03.943132 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/tooz/drivers/etcd3gw.py", line 20, in <module>
Mar 19 16:14:03.943132 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service import etcd3gw
Mar 19 16:14:03.943132 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service ModuleNotFoundError: No module named 'etcd3gw'
Mar 19 16:14:03.943132 ubuntu-bionic-rax-dfw-0003973610 cinder-backup[28520]: ERROR oslo_service.service
http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Error%20starting%20thread.:%20ModuleNotFoundError:%20No%20module%20named%20'etcd3gw'%5C%22%20AND%20tags:%5C%22screen%5C%22&from=10d
This only shows up in the grenade-py3 job, so I wonder if https://github.com/openstack-dev/grenade/commit/d5dc033a2e6df988b0576fc0417be46cc8d157ed is somehow involved.
or maybe this https://github.com/openstack-dev/devstack/commit/676957ffcff78e790134776f71035a3b14974896
In that job above that failed, etcd3gw is not in the py3 output:
http:// logs.openstack. org/96/ 644596/ 1/check/ grenade- py3/c5059ab/ logs/pip3- freeze. txt.gz
But it is in the py2 output:
http:// logs.openstack. org/96/ 644596/ 1/check/ grenade- py3/c5059ab/ logs/pip2- freeze. txt.gz
etcd3gw==0.2.4