n-cpu dies when started before the ironic service is running

[Lucas Alvares Gomes]

Starting n-cpu with the Ironic driver loaded when the Ironic service is not yet running will kill n-cpu.

2014-04-18 14:47:46.221 ERROR ironic.nova.virt.ironic.client_wrapper [req-5dca9434-88c7-446c-aad5-46db38345597 None None] Error contacting Ironic server for 'node.list'. Attempt 5 of 5
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 99, in wait
    writers.get(fileno, noop).cb(fileno)
  File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/nova/nova/openstack/common/service.py", line 480, in run_service
    service.start()
  File "/opt/stack/nova/nova/service.py", line 163, in start
    self.manager.init_host()
  File "/opt/stack/nova/nova/compute/manager.py", line 1035, in init_host
    self._destroy_evacuated_instances(context)
  File "/opt/stack/nova/nova/compute/manager.py", line 690, in _destroy_evacuated_instances
    local_instances = self._get_instances_on_driver(context, filters)
  File "/opt/stack/nova/nova/compute/manager.py", line 668, in _get_instances_on_driver
    driver_instances = self.driver.list_instances()
  File "/opt/stack/ironic/ironic/nova/virt/ironic/driver.py", line 268, in list_instances
    node_list = icli.call("node.list")
  File "/opt/stack/ironic/ironic/nova/virt/ironic/client_wrapper.py", line 96, in call
    raise exception.NovaException(msg)
NovaException: Error contacting Ironic server for 'node.list'. Attempt 5 of 5
Removing descriptor: 6
2014-04-18 14:47:46.222 ERROR nova.openstack.common.threadgroup [-] Error contacting Ironic server for 'node.list'. Attempt 5 of 5
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 117, in wait
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup x.wait()
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 49, in wait
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/service.py", line 480, in run_service
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup service.start()
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/service.py", line 163, in start
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup self.manager.init_host()
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 1035, in init_host
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup self._destroy_evacuated_instances(context)
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 690, in _destroy_evacuated_instances
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup local_instances = self._get_instances_on_driver(context, filters)
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 668, in _get_instances_on_driver
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup driver_instances = self.driver.list_instances()
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup File "/opt/stack/ironic/ironic/nova/virt/ironic/driver.py", line 268, in list_instances
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup node_list = icli.call("node.list")
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup File "/opt/stack/ironic/ironic/nova/virt/ironic/client_wrapper.py", line 96, in call
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup raise exception.NovaException(msg)
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup NovaException: Error contacting Ironic server for 'node.list'. Attempt 5 of 5
2014-04-18 14:47:46.222 TRACE nova.openstack.common.threadgroup
[stack@localhost devstack]$

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/88590

[aeva black (tenbrae)]

Updating the status as the previously linked patch was abandoned.

[aeva black (tenbrae)]

As far as I can tell, this bug was fixed by a combination of patches to the nova.virt.ironic driver, including the client wrapper's retry logic. It does not seem to affect n-cpu today so I am marking it fixed.

Please re-open if you can confirm this bug is still a problem.