All OVB job in master periodic pipeline are failing to load ilo driver during openstack overcloud node import instackenv.json

Bug #1818202 reported by Gabriele Cerami on 2019-03-01
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Gabriele Cerami

Bug Description

All periodic OVB jobs in master show a log similar to this:

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/7bcad2a/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2019-03-01_01_21_02

2019-03-01 01:21:02 | Exception occured while running the command
2019-03-01 01:21:02 | Traceback (most recent call last):
2019-03-01 01:21:02 | File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 29, in run
2019-03-01 01:21:02 | super(Command, self).run(parsed_args)
2019-03-01 01:21:02 | File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2019-03-01 01:21:02 | return super(Command, self).run(parsed_args)
2019-03-01 01:21:02 | File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
2019-03-01 01:21:02 | return_code = self.take_action(parsed_args) or 0
2019-03-01 01:21:02 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_node.py", line 284, in take_action
2019-03-01 01:21:02 | instance_boot_option=parsed_args.instance_boot_option
2019-03-01 01:21:02 | File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/baremetal.py", line 80, in register_or_update
2019-03-01 01:21:02 | 'Exception registering nodes: {}'.format(payload['message']))
2019-03-01 01:21:02 | RegisterOrUpdateError: Exception registering nodes: Resource temporarily unavailable, please retry. (HTTP 503)
2019-03-01 01:21:02 | Exception registering nodes: Resource temporarily unavailable, please retry. (HTTP 503)
2019-03-01 01:21:02 | Resource temporarily unavailable, please retry. (HTTP 503)

Mistral logs at

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/d6d092d/logs/undercloud/var/log/containers/mistral/executor.log.txt.gz#_2019-03-01_01_05_46_971

show

2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal [req-d5f2acc5-9e93-4f11-94de-a65878c8893c 237656d81d3c45e2853ef02d1cf1ac12 f484b8ae54484d5ebbe6ea7d19f77372 - default default] Error registering nodes with ironic.: ServiceUnavailable: Resource temporarily unavailable, please retry. (HTTP 503)
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal Traceback (most recent call last):
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal File "/usr/lib/python2.7/site-packages/tripleo_common/actions/baremetal.py", line 73, in run
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal ramdisk_name=self.ramdisk_name)
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal File "/usr/lib/python2.7/site-packages/tripleo_common/utils/nodes.py", line 591, in register_all_nodes
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal node = _update_or_register_ironic_node(node, node_map, client=client)
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal File "/usr/lib/python2.7/site-packages/tripleo_common/utils/nodes.py", line 546, in _update_or_register_ironic_node
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal ironic_node = register_ironic_node(node, client)
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal File "/usr/lib/python2.7/site-packages/tripleo_common/utils/nodes.py", line 423, in register_ironic_node
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal ironic_node = client.node.create(**create_map)
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal File "/usr/lib/python2.7/site-packages/ironicclient/common/base.py", line 278, in create
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal resp, body = self.api.json_request('POST', url, body=new)
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 496, in json_request
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal resp, body_iter = self._http_request(url, method, **kwargs)
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 287, in wrapper
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal return func(self, url, method, **kwargs)
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 477, in _http_request
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal error_json.get('debuginfo'), method, url)
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal ServiceUnavailable: Resource temporarily unavailable, please retry. (HTTP 503)
2019-03-01 01:05:46.971 8 ERROR tripleo_common.actions.baremetal

Ironic logs repeatedly show this error

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-master/15e724f/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz#_2019-03-01_01_28_03_988

2019-03-01 01:28:03.988 10 ERROR oslo_service.service [req-a798bd9b-57ce-42b5-a0d5-2ef17b35de7a - - - - -] Error starting thread.: DriverLoadError: Driver, hardware type or interface ilo could not be loaded. Reason: 'module' object has no attribute 'noValue'.
2019-03-01 01:28:03.988 10 ERROR oslo_service.service Traceback (most recent call last):
2019-03-01 01:28:03.988 10 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 796, in run_service
2019-03-01 01:28:03.988 10 ERROR oslo_service.service service.start()
2019-03-01 01:28:03.988 10 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/common/rpc_service.py", line 61, in start
2019-03-01 01:28:03.988 10 ERROR oslo_service.service self.manager.init_host(admin_context)
2019-03-01 01:28:03.988 10 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/conductor/base_manager.py", line 128, in init_host
2019-03-01 01:28:03.988 10 ERROR oslo_service.service hardware_types = driver_factory.hardware_types()
2019-03-01 01:28:03.988 10 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 253, in hardware_types
2019-03-01 01:28:03.988 10 ERROR oslo_service.service return _get_all_drivers(HardwareTypesFactory())
2019-03-01 01:28:03.988 10 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 323, in __init__
2019-03-01 01:28:03.988 10 ERROR oslo_service.service self.__class__._init_extension_manager()
2019-03-01 01:28:03.988 10 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2019-03-01 01:28:03.988 10 ERROR oslo_service.service return f(*args, **kwargs)
2019-03-01 01:28:03.988 10 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 393, in _init_extension_manager
2019-03-01 01:28:03.988 10 ERROR oslo_service.service on_missing_entrypoints_callback=missing_callback))
2019-03-01 01:28:03.988 10 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 81, in __init__
2019-03-01 01:28:03.988 10 ERROR oslo_service.service verify_requirements)
2019-03-01 01:28:03.988 10 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 203, in _load_plugins
2019-03-01 01:28:03.988 10 ERROR oslo_service.service self._on_load_failure_callback(self, ep, err)
2019-03-01 01:28:03.988 10 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 378, in _catch_driver_not_found
2019-03-01 01:28:03.988 10 ERROR oslo_service.service raise exception.DriverLoadError(driver=ep.name, reason=exc)
2019-03-01 01:28:03.988 10 ERROR oslo_service.service DriverLoadError: Driver, hardware type or interface ilo could not be loaded. Reason: 'module' object has no attribute 'noValue'.
2019-03-01 01:28:03.988 10 ERROR oslo_service.service

Tags: ci Edit Tag help
Gabriele Cerami (gcerami) wrote :

This was caused by a bump in the version of pysnmp.

pysnmp has a soft requirement on pyasn. Bumping the version made it incompatible with the version of pyasn available, and wasn't detected immediately in the dependencies outside of openstack.
pysnmp version has now been reverted, and next run should not fail on this error anymore.

Details are here

https://review.rdoproject.org/r/19060

Changed in tripleo:
status: Triaged → Fix Committed
wes hayutin (weshayutin) on 2019-03-01
Changed in tripleo:
status: Fix Committed → Fix Released
status: Fix Released → Fix Committed
Rafael Folco (rafaelfolco) wrote :

Not seeing this in ovb jobs anymore. Closing. Please re-open if necessary.

Although recent jobs did not pass, they fail for a different reason.

- Checked ironic-conductor logs - no errors
- Checked overcloud prep images logs - no errors
- Checked mistral logs - no errors

Last run failed on a different error:

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/e5e364b/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz

tags: removed: promotion-blocker
Marios Andreou (marios-b) wrote :

(marios|ruck)
& confirmed latest run is timing out but the bug in the description is no longer seen http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/1c14434/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz

2019-03-18 00:35:08 | Successfully registered node UUID 4e2a0776-48cc-4e06-ae18-b1ec6341d2dc
2019-03-18 00:35:08 | Successfully registered node UUID 2bf7fedc-46f4-485d-bc8d-226423d2a3e7
2019-03-18 00:35:08 | Successfully registered node UUID df872c72-f9b0-49b7-9882-92a4f7a3a613
2019-03-18 00:35:08 | Successfully registered node UUID 4f6a35cc-5972-484f-a300-ba3fcedbb9b3

wes hayutin (weshayutin) on 2019-03-18
Changed in tripleo:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers