centos-8 master tripleo component tests are failing with "ModuleNotFoundError: No module named 'blazarclient'"

[Ronelle Landy]

Tests running on master in the openstack-component-tripleo pipeline are all failing with:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 195, in _load_plugins
    verify_requirements,
  File "/usr/lib/python3.6/site-packages/stevedore/enabled.py", line 79, in _load_one_plugin
    verify_requirements,
  File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 223, in _load_one_plugin
    plugin = ep.resolve()
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3.6/site-packages/heat/engine/clients/os/blazar.py", line 14, in <module>
    from blazarclient import client as blazar_client
ModuleNotFoundError: No module named 'blazarclient'

Example logs:

https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-tripleo-master/0a97b68/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-master/d575aed/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

[Ronelle Landy (rlandy)]

Possibly related to https://opendev.org/openstack/heat/commit/2aea5c9b3224215e246b7508b5f81a7d36cf1abd?

[Ronelle Landy (rlandy)]

Thanks for Sandeep and Yatin for the pointers here ...
requires a clients promotion.
Assigning to ruck/rovers to get the right versions in

[Rabi Mishra (rabi)]

I think delorean-component-clients needs promotion with stevedore 3.1.0
heat has bumped its requirement in the above mentioned change.

These jobs use:

2020-07-16 19:44:33 | python3-stevedore-2.0.1-0.20200630174956.01c12ec.el8.noarch

[Sandeep Yadav (sandeepyadav93)]

Hello Rabi/Team,

We have promoted stevedore but still integration line jobs are failing:-

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/ee6da98/logs/undercloud/var/log/containers/heat/heat-engine.log.txt.gz

~~~
2020-07-20 01:52:11.230 7 CRITICAL heat-engine [-] Unhandled error: AttributeError: 'EntryPoint' object has no attribute 'module'
2020-07-20 01:52:11.230 7 ERROR heat-engine Traceback (most recent call last):
2020-07-20 01:52:11.230 7 ERROR heat-engine File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 204, in _load_plugins
2020-07-20 01:52:11.230 7 ERROR heat-engine verify_requirements,
2020-07-20 01:52:11.230 7 ERROR heat-engine File "/usr/lib/python3.6/site-packages/stevedore/enabled.py", line 79, in _load_one_plugin
2020-07-20 01:52:11.230 7 ERROR heat-engine verify_requirements,
2020-07-20 01:52:11.230 7 ERROR heat-engine File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 234, in _load_one_plugin
2020-07-20 01:52:11.230 7 ERROR heat-engine plugin = ep.load()
2020-07-20 01:52:11.230 7 ERROR heat-engine File "/usr/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 92, in load
2020-07-20 01:52:11.230 7 ERROR heat-engine module = import_module(match.group('module'))
2020-07-20 01:52:11.230 7 ERROR heat-engine File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
2020-07-20 01:52:11.230 7 ERROR heat-engine return _bootstrap._gcd_import(name[level:], package, level)
2020-07-20 01:52:11.230 7 ERROR heat-engine File "<frozen importlib._bootstrap>", line 994, in _gcd_import
2020-07-20 01:52:11.230 7 ERROR heat-engine File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2020-07-20 01:52:11.230 7 ERROR heat-engine File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2020-07-20 01:52:11.230 7 ERROR heat-engine File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
2020-07-20 01:52:11.230 7 ERROR heat-engine File "<frozen importlib._bootstrap_external>", line 678, in exec_module
2020-07-20 01:52:11.230 7 ERROR heat-engine File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2020-07-20 01:52:11.230 7 ERROR heat-engine File "/usr/lib/python3.6/site-packages/heat/engine/clients/os/blazar.py", line 14, in <module>
2020-07-20 01:52:11.230 7 ERROR heat-engine from blazarclient import client as blazar_client
2020-07-20 01:52:11.230 7 ERROR heat-engine ModuleNotFoundError: No module named 'blazarclient'
2020-07-20 01:52:11.230 7 ERROR heat-engine
~~~

Package version:-

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/ee6da98/logs/undercloud/var/log/extra/rpm-list.txt.gz

stevedore package version:-
~~~
python3-stevedore-3.1.0-0.20200717071436.bc04ca9.el8.noarch
~~~

Heat package version:-
~~~
heat-cfntools-1.4.2-11.el8.noarch
openstack-heat-agents-2.1.0-0.20200619172427.0e188d0.el8.noarch
openstack-heat-api-14.1.0-0.2020...

[Sandeep Yadav (sandeepyadav93)]

Hello,

Earlier, job was failing during standalone deploy, now its failing at later stage in tempest run - TASK [os_tempest : List tempest tests].

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/ee6da98/job-output.txt

But heat-engine have same error logs:-

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/ee6da98/logs/undercloud/var/log/containers/heat/heat-engine.log.txt.gz
~~~
2020-07-20 01:52:11.230 7 ERROR heat-engine ModuleNotFoundError: No module named 'blazarclient
~~~

It seems heat-launcher is working fine but not heat-engine.

Fyi...

To promote tripleo/clients components line(together) we used test patch[1]. Components jobs (tripleo/clients) are expected to fail until there is promotion to current-tripleo.

To promote, tripleo-ci-testing to current-tripleo we need integration jobs to pass but currently they are failing . stevedore3.1 is already present in tripleo-ci-testing[2](current hash [3]).

[1] https://review.rdoproject.org/r/28586.
[2] https://trunk.rdoproject.org/centos8-master/tripleo-ci-testing/delorean.repo
[3] https://trunk.rdoproject.org/centos8/component/clients/bc/04/bc04ca91e07b067bcd19d5d9ba4cdc61b0d7bd2f_3c8dd656/

[yatin (yatinkarel)]

Proposed https://review.rdoproject.org/r/#/c/28604/ to update importlib-metadata and resources. This should likely clear the issue with heat-engine container.

latest heat and stevedore-3.1.0 was tested with https://logserver.rdoproject.org/29/28529/2/experimental/distgit-tripleo-master-testing-centos-8-standalone-scenario001/0785157/logs/undercloud/var/log/extra/package-list-installed.txt.gz, and it passed both heat-launcher and heat-engine container phase. But due to other bug https://bugs.launchpad.net/tripleo/+bug/1886914 containers were not updated and thus failure didn't triggered and job resulted in false positive.