Ironic

unprovision attempts to validate driver state

Bug #1345334 reported by Robert Collins on 2014-07-20

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Ironic	Fix Released	High	Ghe Rivero	Ironic 2014.2 "juno"

Bug Description

This means that a failed node may be un-unprovisionable:

2014-07-20 02:16:17.196 4721 ERROR nova.compute.manager [req-1c46b342-8b79-442b-8d84-54b6a7675a6a None] [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] Setting instance vm_state to ERROR
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] Traceback (most recent call last):
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2352, in do_terminate_inst
ance
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] self._delete_instance(context, instance, bdms, quotas)
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/hooks.py", line 131, in inner
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] rv = f(*args, **kwargs)
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2321, in _delete_instance
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] quotas.rollback()
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] six.reraise(self.type_, self.value, self.tb)
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2296, in _delete_instance
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] self._shutdown_instance(context, instance, bdms)
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2238, in _shutdown_instanc
e
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] requested_networks)
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] six.reraise(self.type_, self.value, self.tb)
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2227, in _shutdown_instanc
e
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] block_device_info)
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironic/nova/virt/ironic/driver.py", line 624, in destroy
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] self._unprovision(icli, instance, node)
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironic/nova/virt/ironic/driver.py", line 559, in _unprovision
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] icli.call("node.set_provision_state", node.uuid, "deleted")
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironic/nova/virt/ironic/client_wrapper.py", line 89, in call
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] return self._multi_getattr(client, method)(*args, **kwargs)
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironicclient/v1/node.py", line 102, in set_provision_state
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] return self._update(self._path(path), target, method='PUT')
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironicclient/common/base.py", line 70, in _update
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] resp, body = self.api.json_request(method, url, body=body)
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line 188, in json_request
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] resp, body_iter = self._http_request(url, method, **kwargs)
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line 171, in _http_request
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] error_json.get('debuginfo'), method, url)
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207] InternalServerError: RPC do_node_tear_down failed to validate deploy info. Error: Can not validate PXE bootloader. The following parameters were not passed to ironic: ['root_gb', 'image_source'] (HTTP 500)
2014-07-20 02:16:17.196 4721 TRACE nova.compute.manager [instance: 60b4328d-ca2a-42d7-a6c8-34a1a8594207]

Robert Collins (lifeless) on 2014-07-20

Changed in ironic:
status:	New → Triaged
importance:	Undecided → High

Revision history for this message

Robert Collins (lifeless) wrote on 2014-07-20:

ironic node-show 1b44fa97-a41f-4aa1-a578-08e82b6ed624
+------------------------+--------------------------------------------------------------------------+
| Property               | Value                                                                    |
+------------------------+--------------------------------------------------------------------------+
| instance_uuid          | 60b4328d-ca2a-42d7-a6c8-34a1a8594207                                     |
| target_power_state     | None                                                                     |
| properties             | {u'memory_mb': u'98304', u'cpu_arch': u'amd64', u'local_gb': u'1600',    |
|                        | u'cpus': u'24'}                                                          |
| maintenance            | False                                                                    |
| driver_info            | {u'pxe_deploy_ramdisk': u'eab3021b-1511-4617-ba0b-48658f4eb52d',         |
|                        | u'pxe_deploy_kernel': u'4a9d6fa2-4e1e-4ed7-844d-51b037545681',           |
|                        | u'ipmi_address': u'xxxxx', u'ipmi_username': u'Administrator',     |
|                        | u'ipmi_password': u'xxxx'}                                           |
| extra                  | {}                                                                       |
| last_error             | Failed to tear down. Error: Failed to set node power state to power off. |
| created_at             | 2014-07-08T23:09:05+00:00                                                |
| target_provision_state | None                                                                     |
| driver                 | pxe_ipmitool                                                             |
| updated_at             | 2014-07-20T02:20:00+00:00                                                |
| instance_info          | {}                                                                       |
| chassis_uuid           | None                                                                     |
| provision_state        | error                                                                    |
| reservation            | None                                                                     |
| power_state            | power on                                                                 |
| console_enabled        | False                                                                    |
| uuid                   | 1b44fa97-a41f-4aa1-a578-08e82b6ed624                                     |
+------------------------+--------------------------------------------------------------------------+

Revision history for this message

Ghe Rivero (ghe.rivero) wrote on 2014-07-21:

Related bug: https://bugs.launchpad.net/ironic/+bug/1335909

Changed in ironic:
assignee:	nobody → Ghe Rivero (ghe.rivero)

OpenStack Infra (hudson-openstack) on 2014-07-21

Changed in ironic:
status:	Triaged → In Progress

aeva black (tenbrae) on 2014-08-11

Changed in ironic:
milestone:	none → juno-3

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-18: Fix merged to ironic (master)

Reviewed: https://review.openstack.org/103685
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=e10920e9a783cb6aae43bcb2c31b7cfbbfaa9904
Submitter: Jenkins
Branch: master

commit e10920e9a783cb6aae43bcb2c31b7cfbbfaa9904
Author: Ghe Rivero <email address hidden>
Date: Mon Jun 30 23:18:40 2014 +0000

Fix tear_down a node with missing info

    When tearing down a node, there could be some situations where the
    instance info is missing some properties (image source deleted before
    tearing down the node), so skipping node info validation. Instead,
    perform a power validation to assure that the node can be power
    managed.

    Closes-Bug: #1335909
    Closes-Bug: #1345334
    Change-Id: Idb1ca720bbd19de1fccfe049e592136dbca2aacf

Changed in ironic:
status:	In Progress → Fix Committed

Revision history for this message

Davide Guerri (davide-guerri) wrote on 2014-08-19:

I have just tested the master tip and this issue seems be fixed now.

Thanks Ghe!

Thierry Carrez (ttx) on 2014-09-04

Changed in ironic:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-10-16

Changed in ironic:
milestone:	juno-3 → 2014.2

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.