Node instance_info not updated during rebuild, rebuilds do not deploy new images

Bug #1334905 reported by Ghe Rivero
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Fix Released
High
Adam Gandelman

Bug Description

When a image is recreated (updated) and the node needs to be redeployed via 'heat stack-update', it fails. The error shows that the ironic node is still refering to the old image. This should be updated to use the new one. (image_source)

root@undercloud-undercloud-dudmy7r74vmg:/mnt/state/var/lib/mysql# ironic node-show 1194694b-5657-4f4d-9456-8ebc2a98bee2
+------------------------+--------------------------------------------------------------------+
| Property | Value |
+------------------------+--------------------------------------------------------------------+
| instance_uuid | dd5f0437-1f77-471f-b24b-e1d6c1dfc41c |
| target_power_state | None |
| properties | {u'memory_mb': u'4096', u'cpu_arch': u'amd64', u'local_gb': u'20', |
| | u'cpus': u'2'} |
| maintenance | False |
| driver_info | {u'pxe_deploy_ramdisk': u'644d7835-152e-4206-b2bc-6e54f8e71201', |
| | u'pxe_deploy_kernel': u'696a2fa5-8b3e-4ff7-bed5-15dd8b109069', |
| | u'ssh_username': u'ghe', u'ssh_key_filename': |
| | u'/mnt/state/var/lib/ironic/virtual-power-key', u'ssh_address': |
| | u'192.168.122.1', u'ssh_virt_type': u'virsh'} |
| extra | {} |
| last_error | None |
| created_at | 2014-06-26T22:33:04+00:00 |
| target_provision_state | None |
| driver | pxe_ssh |
| updated_at | 2014-06-27T00:11:04+00:00 |
| instance_info | {u'ramdisk': u'397b7985-df56-4487-ac39-139b11481193', u'kernel': |
| | u'63a95d97-af73-42ab-a4e8-d7531d74def3', u'root_gb': u'10', |
| | u'image_source': u'fa8d3f4b-c8ac-422a-9989-3477a5ef8080', |
| | u'ephemeral_format': u'ext4', u'ephemeral_gb': u'10', |
| | u'preserve_ephemeral': u'True', u'deploy_key': |
| | u'TUZEAXCR6AUUFD3RC2RPHERBZTYRM9LI', u'swap_mb': u'0'} |
| chassis_uuid | None |
| provision_state | active |
| reservation | None |
| power_state | power on |
| console_enabled | False |
| uuid | 1194694b-5657-4f4d-9456-8ebc2a98bee2 |
+------------------------+--------------------------------------------------------------------+

root@undercloud-undercloud-dudmy7r74vmg:/mnt/state/var/lib/mysql# nova show dd5f0437-1f77-471f-b24b-e1d6c1dfc41c
| fault | {"message": "Failed to request Ironic to rebuild instance dd5f0437-1f77-471f-b24b-e1d6c1dfc41c: RPC do_node_deploy failed to validate deploy info. Error: Image fa8d3f4b-c8ac-422a-9989-3477a5ef8080 is missing the following properties: kernel_id, ramdisk_id (HTTP 500)", "code": 400, "created": "2014-06-26T23:42:59Z"} |

root@undercloud-undercloud-dudmy7r74vmg:/mnt/state/var/lib/mysql# glance image-show fa8d3f4b-c8ac-422a-9989-3477a5ef8080
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 8dfad757e3ad9fb6cc7df2776a0a3cff |
| container_format | bare |
| created_at | 2014-06-26T22:42:07 |
| deleted | True |
| deleted_at | 2014-06-26T23:39:14 |
| disk_format | qcow2 |
| id | fa8d3f4b-c8ac-422a-9989-3477a5ef8080 |
| is_public | True |
| min_disk | 0 |
| min_ram | 0 |
| name | overcloud-control |
| owner | 88e6e44de1a94da5ba84773ec99311c5 |
| protected | False |
| size | 1287204352 |
| status | deleted |
| updated_at | 2014-06-26T23:39:14 |
+------------------+--------------------------------------+
root@undercloud-undercloud-dudmy7r74vmg:/mnt/state/var/lib/mysql# glance image-show overcloud-control
+-----------------------+--------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------+
| Property 'kernel_id' | 145efc1a-f4e2-4f05-a895-feadebb7d943 |
| Property 'ramdisk_id' | 704c76ad-cfd4-47c8-901c-661816b5f530 |
| checksum | a68a48d573d892d323c6723b2301d2ab |
| container_format | bare |
| created_at | 2014-06-26T23:39:16 |
| deleted | False |
| disk_format | qcow2 |
| id | 8b570f64-2f90-4595-ae5f-82ee41147ebd |
| is_public | True |
| min_disk | 0 |
| min_ram | 0 |
| name | overcloud-control |
| owner | 88e6e44de1a94da5ba84773ec99311c5 |
| protected | False |
| size | 1287696896 |
| status | active |
| updated_at | 2014-06-26T23:39:20 |
+-----------------------+--------------------------------------+

A possible solution could be to test if the image is in state deleted, and update the image_source based in the name.

Tags: nova-driver
Revision history for this message
Adam Gandelman (gandelman-a) wrote :

TripleO + heat were used to produce the original bug. Before rebuild, the original image is deleted hence the validation failing during re-deployment. But the bug is more fundamental, the current rebulid code only updates the preserve_ephemeral flag, leaving image related driver fields untouched. The rebuilt server ends up rebuilding (or attempting to) with the original image information, regardless of what user image (if any) was passed during nova rebuild.

Changed in ironic:
status: New → Confirmed
summary: - Node image_source not updated
+ Node image_source not updated, rebuilds do not deploy new images
Dmitry Tantsur (divius)
summary: - Node image_source not updated, rebuilds do not deploy new images
+ Node instance_info not updated during rebuild, rebuilds do not deploy
+ new images
Changed in ironic:
importance: Undecided → High
tags: added: nova-driver
Changed in ironic:
status: Confirmed → Triaged
Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Not sure why there is no linkage. Patch proposed @ https://review.openstack.org/#/c/103227/

Changed in ironic:
assignee: Ghe Rivero (ghe.rivero) → Adam Gandelman (gandelman-a)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/103665

Changed in ironic:
assignee: Adam Gandelman (gandelman-a) → Ghe Rivero (ghe.rivero)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ironic (master)

Change abandoned by Ghe Rivero (<email address hidden>) on branch: master
Review: https://review.openstack.org/103665
Reason: supersede by I6c8aacd36e0095d0590a1dc552730f2d52036225

Revision history for this message
Ghe Rivero (ghe.rivero) wrote :
aeva black (tenbrae)
Changed in ironic:
milestone: none → juno-2
Changed in ironic:
assignee: Ghe Rivero (ghe.rivero) → Adam Gandelman (gandelman-a)
Revision history for this message
Adam Gandelman (gandelman-a) wrote :
Changed in ironic:
status: In Progress → Fix Committed
Changed in ironic:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ironic:
milestone: juno-2 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.