Ironic

n-cpu cannot be restarted with running instances

Bug #1326988 reported by Adam Gandelman on 2014-06-05

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Ironic	Fix Released	High	Adam Gandelman	Ironic 2014.2 "juno"

Bug Description

If an instance has been spawned using the ironic driver and nova-compute restarts, it fails to start again:

cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf
2014-06-05 15:38:50.213 15658 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2014-06-05 15:38:50.232 15658 INFO nova.virt.driver [-] Loading compute driver 'ironic.nova.virt.ironic.IronicDriver'
2014-06-05 15:38:50.259 15658 INFO oslo.messaging._drivers.impl_rabbit [req-0321d7b0-fb55-41cf-bb30-e6ee2d5c0cd5 ] Connected to AMQP server on 127.0.0.1:5672
2014-06-05 15:38:50.267 15658 INFO oslo.messaging._drivers.impl_rabbit [req-0321d7b0-fb55-41cf-bb30-e6ee2d5c0cd5 ] Connected to AMQP server on 127.0.0.1:5672
2014-06-05 15:38:50.282 15658 AUDIT nova.service [-] Starting compute node (version 2014.2)
2014-06-05 15:38:50.307 15658 INFO urllib3.connectionpool [req-92fbfce3-bb03-49ff-b382-0976191d88d1 ] Starting new HTTP connection (1): 127.0.0.1
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
    readers.get(fileno, noop).cb(fileno)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/nova/nova/openstack/common/service.py", line 490, in run_service
    service.start()
  File "/opt/stack/nova/nova/service.py", line 163, in start
    self.manager.init_host()
  File "/opt/stack/nova/nova/compute/manager.py", line 1048, in init_host
    self._destroy_evacuated_instances(context)
  File "/opt/stack/nova/nova/compute/manager.py", line 684, in _destroy_evacuated_instances
    local_instances = self._get_instances_on_driver(context, filters)
  File "/opt/stack/nova/nova/compute/manager.py", line 652, in _get_instances_on_driver
    driver_uuids = self.driver.list_instance_uuids()
  File "/opt/stack/ironic/ironic/nova/virt/ironic/driver.py", line 333, in list_instance_uuids
    return [i.instance_uuid for i in self.list_instances()]
AttributeError: 'str' object has no attribute 'instance_uuid'
Removing descriptor: 4
2014-06-05 15:38:50.478 15658 ERROR nova.openstack.common.threadgroup [-] 'str' object has no attribute 'instance_uuid'
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 125, in wait
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup x.wait()
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 47, in wait
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/service.py", line 490, in run_service
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup service.start()
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/service.py", line 163, in start
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup self.manager.init_host()
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 1048, in init_host
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup self._destroy_evacuated_instances(context)
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 684, in _destroy_evacuated_instances
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup local_instances = self._get_instances_on_driver(context, filters)
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 652, in _get_instances_on_driver
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup driver_uuids = self.driver.list_instance_uuids()
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup File "/opt/stack/ironic/ironic/nova/virt/ironic/driver.py", line 333, in list_instance_uuids
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup return [i.instance_uuid for i in self.list_instances()]
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup AttributeError: 'str' object has no attribute 'instance_uuid'
2014-06-05 15:38:50.478 15658 TRACE nova.openstack.common.threadgroup

Tags:

Revision history for this message

Adam Gandelman (gandelman-a) wrote on 2014-06-05:

Looks like list_instances() was fixed but not its callers.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-05: Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/98268

Changed in ironic:
assignee:	nobody → Adam Gandelman (gandelman-a)
status:	New → In Progress

Roman Prykhodchenko (romcheg) on 2014-06-06

Changed in ironic:
importance:	Undecided → High

Dmitry Tantsur (divius) on 2014-06-06

tags:

added: driver

aeva black (tenbrae) on 2014-06-09

tags:

added: nova-driver
removed: driver

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-10: Fix merged to ironic (master)

Reviewed: https://review.openstack.org/98268
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=4d4aeab4e57d623ca06bdfdceddbee2791851d03
Submitter: Jenkins
Branch: master

commit 4d4aeab4e57d623ca06bdfdceddbee2791851d03
Author: Adam Gandelman <email address hidden>
Date: Thu Jun 5 15:45:38 2014 -0700

Update Nova driver's list_instance_uuids()

    list_instance_uuid() was dependent on list_instances(). The latter
    was fixed to return the correct results but the former was not,
    causing nova-compute start up failures. This updates it accordingly
    to return the expected output via a client API call.

Change-Id: I27651ca01dbdcc95ca729da7dbbeafdc616eec4e
Closes-bug: #1326988

Changed in ironic:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2014-06-12

Changed in ironic:
milestone:	none → juno-1
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-10-16

Changed in ironic:
milestone:	juno-1 → 2014.2

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.