tripleo

comp-pipeline-network: ModuleNotFoundError: No module named 'os_resource_classes'

Bug #1935974 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Unassigned
tripleo xena-2

Bug Description

In Network component line, all the jobs are failing with following errors: [1]
```
TASK [os_tempest : Ensure private network exists] ******************************
2021-07-12 13:12:43.382545 | primary | Monday 12 July 2021 13:12:43 +0000 (0:00:00.101) 0:35:36.590 ***********
2021-07-12 13:12:45.389296 | primary | FAILED - RETRYING: Ensure private network exists (5 retries left).
2021-07-12 13:12:57.236909 | primary | FAILED - RETRYING: Ensure private network exists (4 retries left).
2021-07-12 13:13:09.015480 | primary | FAILED - RETRYING: Ensure private network exists (3 retries left).
2021-07-12 13:13:20.776635 | primary | FAILED - RETRYING: Ensure private network exists (2 retries left).
2021-07-12 13:13:32.594874 | primary | FAILED - RETRYING: Ensure private network exists (1 retries left).
2021-07-12 13:13:44.314729 | primary | fatal: [undercloud -> 127.0.0.2]: FAILED! => {
2021-07-12 13:13:44.314933 | primary | "attempts": 5,
2021-07-12 13:13:44.314985 | primary | "changed": false,
2021-07-12 13:13:44.315030 | primary | "extra_data": {
2021-07-12 13:13:44.315053 | primary | "data": null,
2021-07-12 13:13:44.315092 | primary | "details": "503 Service Unavailable: No server is available to handle this request.",
2021-07-12 13:13:44.315217 | primary | "response": "<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n"
2021-07-12 13:13:44.315358 | primary | }
2021-07-12 13:13:44.315397 | primary | }
2021-07-12 13:13:44.315412 | primary |
2021-07-12 13:13:44.315422 | primary | MSG:
2021-07-12 13:13:44.315432 | primary |
2021-07-12 13:13:44.315477 | primary | HttpException: 503: Server Error for url: http://192.168.24.3:9696/v2.0/networks?tenant_id=756b1723b8834063832e4e29f6fb311f, 503 Service Unavailable: No server is available to handle this request.
2021-07-12 13:13:44.316860 | primary |
```

By taking a neutron server.log [2]
```
2021-07-12 13:09:34.048 9 DEBUG oslo_concurrency.lockutils [req-7dac5d9d-8203-41b3-919b-192221bdd01c - - - - -] Lock "manager" released by "neutron.manager.NeutronManager._create_instance" :: held 2.593s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-07-12 13:09:34.049 9 ERROR neutron.service [req-7dac5d9d-8203-41b3-919b-192221bdd01c - - - - -] Unrecoverable error: please check log for details.: ModuleNotFoundError: No module named 'os_resource_classes'
2021-07-12 13:09:34.049 9 ERROR neutron.service Traceback (most recent call last):
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/service.py", line 88, in serve_wsgi
2021-07-12 13:09:34.049 9 ERROR neutron.service service.start()
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/service.py", line 64, in start
2021-07-12 13:09:34.049 9 ERROR neutron.service self.wsgi_app = _run_wsgi(self.app_name)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/service.py", line 321, in _run_wsgi
2021-07-12 13:09:34.049 9 ERROR neutron.service app = config.load_paste_app(app_name)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/common/config.py", line 134, in load_paste_app
2021-07-12 13:09:34.049 9 ERROR neutron.service app = loader.load_app(app_name)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/oslo_service/wsgi.py", line 351, in load_app
2021-07-12 13:09:34.049 9 ERROR neutron.service return deploy.loadapp("config:%s" % self.config_path, name=name)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/deploy/loadwsgi.py", line 253, in loadapp
2021-07-12 13:09:34.049 9 ERROR neutron.service return loadobj(APP, uri, name=name, **kw)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/deploy/loadwsgi.py", line 278, in loadobj
2021-07-12 13:09:34.049 9 ERROR neutron.service return context.create()
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/deploy/loadwsgi.py", line 715, in create
2021-07-12 13:09:34.049 9 ERROR neutron.service return self.object_type.invoke(self)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/deploy/loadwsgi.py", line 150, in invoke
2021-07-12 13:09:34.049 9 ERROR neutron.service **context.local_conf)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/deploy/util.py", line 55, in fix_call
2021-07-12 13:09:34.049 9 ERROR neutron.service val = callable(*args, **kw)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/urlmap.py", line 31, in urlmap_factory
2021-07-12 13:09:34.049 9 ERROR neutron.service app = loader.get_app(app_name, global_conf=global_conf)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/deploy/loadwsgi.py", line 356, in get_app
2021-07-12 13:09:34.049 9 ERROR neutron.service name=name, global_conf=global_conf).create()
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/deploy/loadwsgi.py", line 715, in create
2021-07-12 13:09:34.049 9 ERROR neutron.service return self.object_type.invoke(self)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/deploy/loadwsgi.py", line 150, in invoke
2021-07-12 13:09:34.049 9 ERROR neutron.service **context.local_conf)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/deploy/util.py", line 55, in fix_call
2021-07-12 13:09:34.049 9 ERROR neutron.service val = callable(*args, **kw)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/auth.py", line 62, in pipeline_factory
2021-07-12 13:09:34.049 9 ERROR neutron.service app = loader.get_app(pipeline[-1])
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/deploy/loadwsgi.py", line 356, in get_app
2021-07-12 13:09:34.049 9 ERROR neutron.service name=name, global_conf=global_conf).create()
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/deploy/loadwsgi.py", line 715, in create
2021-07-12 13:09:34.049 9 ERROR neutron.service return self.object_type.invoke(self)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/deploy/loadwsgi.py", line 152, in invoke
2021-07-12 13:09:34.049 9 ERROR neutron.service return fix_call(context.object, context.global_conf, **context.local_conf)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/paste/deploy/util.py", line 55, in fix_call
2021-07-12 13:09:34.049 9 ERROR neutron.service val = callable(*args, **kw)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/api/v2/router.py", line 25, in _factory
2021-07-12 13:09:34.049 9 ERROR neutron.service return pecan_app.v2_factory(global_config, **local_config)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/app.py", line 47, in v2_factory
2021-07-12 13:09:34.049 9 ERROR neutron.service startup.initialize_all()
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/startup.py", line 39, in initialize_all
2021-07-12 13:09:34.049 9 ERROR neutron.service manager.init()
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/manager.py", line 301, in init
2021-07-12 13:09:34.049 9 ERROR neutron.service NeutronManager.get_instance()
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/manager.py", line 252, in get_instance
2021-07-12 13:09:34.049 9 ERROR neutron.service cls._create_instance()
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2021-07-12 13:09:34.049 9 ERROR neutron.service return f(*args, **kwargs)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/manager.py", line 238, in _create_instance
2021-07-12 13:09:34.049 9 ERROR neutron.service cls._instance = cls()
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/manager.py", line 132, in __init__
2021-07-12 13:09:34.049 9 ERROR neutron.service self._load_service_plugins()
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/manager.py", line 201, in _load_service_plugins
2021-07-12 13:09:34.049 9 ERROR neutron.service 'neutron.service_plugins', provider)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/manager.py", line 156, in _get_plugin_class
2021-07-12 13:09:34.049 9 ERROR neutron.service return self.load_class_for_provider(namespace, plugin_provider)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/manager.py", line 153, in load_class_for_provider
2021-07-12 13:09:34.049 9 ERROR neutron.service LOG.error("Plugin '%s' not found.", plugin_provider)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-07-12 13:09:34.049 9 ERROR neutron.service self.force_reraise()
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-07-12 13:09:34.049 9 ERROR neutron.service raise self.value
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/manager.py", line 150, in load_class_for_provider
2021-07-12 13:09:34.049 9 ERROR neutron.service plugin_provider)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron_lib/utils/runtime.py", line 114, in load_class_by_alias_or_classname
2021-07-12 13:09:34.049 9 ERROR neutron.service namespace, name, warn_on_missing_entrypoint=False)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/stevedore/driver.py", line 62, in __init__
2021-07-12 13:09:34.049 9 ERROR neutron.service warn_on_missing_entrypoint=warn_on_missing_entrypoint
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/stevedore/named.py", line 81, in __init__
2021-07-12 13:09:34.049 9 ERROR neutron.service verify_requirements)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 233, in _load_plugins
2021-07-12 13:09:34.049 9 ERROR neutron.service self._on_load_failure_callback(self, ep, err)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 225, in _load_plugins
2021-07-12 13:09:34.049 9 ERROR neutron.service verify_requirements,
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/stevedore/named.py", line 158, in _load_one_plugin
2021-07-12 13:09:34.049 9 ERROR neutron.service verify_requirements,
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 255, in _load_one_plugin
2021-07-12 13:09:34.049 9 ERROR neutron.service plugin = ep.load()
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 105, in load
2021-07-12 13:09:34.049 9 ERROR neutron.service module = import_module(match.group('module'))
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
2021-07-12 13:09:34.049 9 ERROR neutron.service return _bootstrap._gcd_import(name[level:], package, level)
2021-07-12 13:09:34.049 9 ERROR neutron.service File "<frozen importlib._bootstrap>", line 994, in _gcd_import
2021-07-12 13:09:34.049 9 ERROR neutron.service File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2021-07-12 13:09:34.049 9 ERROR neutron.service File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2021-07-12 13:09:34.049 9 ERROR neutron.service File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
2021-07-12 13:09:34.049 9 ERROR neutron.service File "<frozen importlib._bootstrap_external>", line 678, in exec_module
2021-07-12 13:09:34.049 9 ERROR neutron.service File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2021-07-12 13:09:34.049 9 ERROR neutron.service File "/usr/lib/python3.6/site-packages/neutron/services/qos/qos_plugin.py", line 39, in <module>
2021-07-12 13:09:34.049 9 ERROR neutron.service import os_resource_classes as orc
2021-07-12 13:09:34.049 9 ERROR neutron.service ModuleNotFoundError: No module named 'os_resource_classes'
```

By comparing with successful run, that is the only error looks suspicious.

https://review.opendev.org/c/openstack/neutron/+/799676 - Use os-resource-classes lib for resource classes names might have caused it.
It needs to be investigated.

Links:
[1]. https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario007-standalone-network-master/85b1d2f/job-output.txt

[2]. https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario007-standalone-network-master/85b1d2f/logs/undercloud/var/log/containers/neutron/server.log.txt.gz

Revision history for this message
chandan kumar (chkumar246) wrote :

Proposed a patch on RDO side to fix import error issue: https://review.rdoproject.org/r/c/openstack/neutron-distgit/+/34515

wes hayutin (weshayutin)
summary: - ModuleNotFoundError: No module named 'os_resource_classes'
+ comp-pipeline-network: ModuleNotFoundError: No module named
+ 'os_resource_classes'
Revision history for this message
wes hayutin (weshayutin) wrote :

network component status=Red last_promotion=2021-07-14 14:03:32
Hash_under_test=https://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_detail.html?commit_hash=3cae410b3094e74549a1941e0b7d833229ae51d5&distro_hash=94be9790e7e0cecc4993eb4de7834b22f9b
3362b
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Jobs which passed: ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ periodic-tripleo-ci-centos-8-scenario007-standalone-network-master │
│ periodic-tripleo-ci-centos-8-scenario010-standalone-network-master │
│ periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-master │
│ periodic-tripleo-ci-centos-8-containers-multinode-network-master │
│ periodic-tripleo-ci-centos-8-standalone-network-master │
└──────────────────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Jobs which failed: ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ periodic-tripleo-ci-centos-8-standalone-network-master-validation │
└──────────────────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Pending running jobs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
└──────────────────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ network component jobs which need pass to promote the hash: ┃ Integration ┃ Integration ┃ Integration ┃
┃ ┃ PASSED History ┃ FAILURE History ┃ Other History ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
└──────────────────────────────────────────────────────────────────────────────────┴─────────────────┴─────────────────┴─────────────────┘
Logs of failing jobs:
https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-network-master-validation/9bd678a

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.