Nova rescheduling failure
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Ironic |
Fix Released
|
High
|
Lucas Alvares Gomes | ||
Bug Description
If something fails in the spawn() method (E.g we Nova Ironic Driver couldn't add the PXE parameters in Ironic) nova will try to rescheduler the instance and as part of the rescheduling process nova will call destroy() on that instance which will then send a request to Ironic to undeploy the node, but the problem is that the node wasn't even deployed, so another error will be raised which will then overwrite the previous one making it very difficult to debug the real cause of the problem. Also the instance can't be deleted from nova after it.
LOGS:
$ nova boot...
stack@stack-
+------
| ID | Name | Status | Task State | Power State | Networks |
+------
| e155d6c7-
+------
stack@stack-
+------
| Property | Value |
+------
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-STS:vm_state | error |
| OS-SRV-
| OS-SRV-
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-03-
| fault | {"message": "RPC do_node_tear_down not allowed for node db2f5627-
| | Traceback (most recent call last): |
| | |
| | File \"/usr/
| | result = f(self, *args, **kwar", "code": 500, "details": " File \"/opt/
| | return function(self, context, *args, **kwargs) |
| | File \"/opt/
| | do_run_instance() |
| | File \"/opt/
| | return f(*args, **kwargs) |
| | File \"/opt/
| | legacy_bdm_in_spec) |
| | File \"/opt/
| | notify(\"error\", fault=e) # notify that build failed |
| | File \"/opt/
| | six.reraise(
| | File \"/opt/
| | instance, image_meta, legacy_bdm_in_spec) |
| | File \"/opt/
| | filter_properties, bdms, legacy_bdm_in_spec) |
| | File \"/opt/
| | self._log_
| | File \"/opt/
| | six.reraise(
| | File \"/opt/
| | bdms, requested_networks) |
| | File \"/opt/
| | requested_networks) |
| | File \"/opt/
| | six.reraise(
| | File \"/opt/
| | block_device_info) |
| | File \"/opt/
| | icli.node.
| | File \"/opt/
| | timer.start(
| | File \"/usr/
| | return hubs.get_
| | File \"/usr/
| | return self.greenlet.
| | File \"/opt/
| | self.f(*self.args, **self.kw) |
| | File \"/opt/
| | func(*args) |
| | File \"/opt/
| | return self._update(
| | File \"/opt/
| | resp, body = self.api.
| | File \"/opt/
| | resp, body_iter = self._http_
| | File \"/opt/
| | error_json.
| | ", "created": "2014-03-
| flavor | baremetal (ae187c1a-
| hostId | ea888db9bf4d0ee
| id | e155d6c7-
| image | cirros-
| key_name | default |
| metadata | {} |
| name | ironictest |
| os-extended-
| public network | 172.24.4.4 |
| status | ERROR |
| tenant_id | 6fa62fa758e6456
| updated | 2014-03-
| user_id | be6a18d4c46c452
+------
2014-03-31 14:11:46.580 13461 ERROR wsme.api [-] Server-side error: "RPC do_node_tear_down not allowed for node 632244aa-
Traceback (most recent call last):
File "/opt/stack/
return func(*args, **kwargs)
File "/opt/stack/
task.
File "/opt/stack/
six.
File "/opt/stack/
% {'node': node_id, 'state': node.provision_
InstanceDeployF
". Detail:
Traceback (most recent call last):
File "/usr/local/
result = f(self, *args, **kwargs)
File "/opt/stack/
pecan.
File "/opt/stack/
topic=topic or self.topic)
File "/opt/stack/
result = rpc.call(context, real_topic, msg, timeout)
File "/opt/stack/
return _get_impl(
File "/opt/stack/
rpc_
File "/opt/stack/
rv = list(rv)
File "/opt/stack/
raise result
InstanceDeployF
Traceback (most recent call last):
File "/opt/stack/
return func(*args, **kwargs)
File "/opt/stack/
task.
File "/opt/stack/
six.
File "/opt/stack/
% {'node': node_id, 'state': node.provision_
InstanceDeployF
| Changed in ironic: | |
| assignee: | nobody → Lucas Alvares Gomes (lucasagomes) |
| importance: | Undecided → High |
| milestone: | none → juno-1 |
| description: | updated |
| Dmitry Tantsur (divius) wrote | #1 |
| Changed in ironic: | |
| status: | New → Triaged |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to ironic (master) | #2 |
| Changed in ironic: | |
| status: | Triaged → In Progress |
| tags: | added: icehouse-backport-potential |
| Changed in ironic: | |
| assignee: | Lucas Alvares Gomes (lucasagomes) → Dmitry "Divius" Tantsur (divius) |
| Changed in ironic: | |
| assignee: | Dmitry "Divius" Tantsur (divius) → Lucas Alvares Gomes (lucasagomes) |
| OpenStack Infra (hudson-openstack) wrote Fix merged to ironic (master) | #3 |
| Changed in ironic: | |
| status: | In Progress → Fix Committed |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to ironic (milestone-proposed) | #4 |
| Changed in ironic: | |
| milestone: | juno-1 → icehouse-rc2 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to ironic (milestone-proposed) | #5 |
| Changed in ironic: | |
| status: | Fix Committed → Fix Released |
| Changed in ironic: | |
| milestone: | icehouse-rc2 → 2014.1 |
My steps to reproduce:
1. Put nodes to maintenance mode
2. Try to deploy anything