rebuild volume-backed instance with image does not replace image in root disk

Bug #1482040 reported by Zhenyu Zheng on 2015-08-06
62
This bug affects 10 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Unassigned

Bug Description

When rebuild an volume-backed instance using normal images from glance,
nothing will be changed for the instance after rebuild except the image name
will be added to the instance, the instance is actually again built using the same
volume.

root@zheng-dev1:/var/log/nova# nova boot --flavor 1 --boot-volume daaddb77-4257-4ccd-86f2-220b31a0ce9b --nic net-id=8744ee96-7690-43bb-89b4-fcac805557bc test_rebuild
+--------------------------------------+--------------------------------------------------+
| Property | Value |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-0000001a |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | KX5f2qfJBs9S |
| config_drive | |
| created | 2015-08-06T04:09:06Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | 2f91193c-c462-4bcf-ad87-692c3d263cfd |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| metadata | {} |
| name | test_rebuild |
| os-extended-volumes:volumes_attached | [{"id": "daaddb77-4257-4ccd-86f2-220b31a0ce9b"}] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | b8112a8d8227490eba99419b8a8c2555 |
| updated | 2015-08-06T04:09:06Z |
| user_id | ed64bccd0227444fa02dbd7695769a7d |
+--------------------------------------+--------------------------------------------------+

root@zheng-dev1:/var/log/nova# nova rebuild 2f91193c-c462-4bcf-ad87-692c3d263cfd 7bdff9a3-d051-4e75-bcd3-de69dbffe063
+-------------------+----------------------------------------------------------------+
| Property | Value |
+-------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | GParrEjj3DEW |
| created | 2015-08-06T04:09:06Z |
| flavor | m1.tiny (1) |
| hostId | e9701452373fee3a5897683c6b914f4ba33b38b0fcff3b3719615bbd |
| id | 2f91193c-c462-4bcf-ad87-692c3d263cfd |
| image | cirros-0.3.4-x86_64-uec (7bdff9a3-d051-4e75-bcd3-de69dbffe063) |
| metadata | {} |
| name | test_rebuild |
| progress | 0 |
| public network | 2001:db8::19, 172.24.4.24 |
| status | REBUILD |
| tenant_id | b8112a8d8227490eba99419b8a8c2555 |
| updated | 2015-08-06T04:09:50Z |
| user_id | ed64bccd0227444fa02dbd7695769a7d |
+-------------------+----------------------------------------------------------------+

root@zheng-dev1:/var/log/nova# nova show 2f91193c-c462-4bcf-ad87-692c3d263cfd
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | zheng-dev1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | zheng-dev1 |
| OS-EXT-SRV-ATTR:instance_name | instance-0000001a |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2015-08-06T04:10:00.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | True |
| created | 2015-08-06T04:09:06Z |
| flavor | m1.tiny (1) |
| hostId | e9701452373fee3a5897683c6b914f4ba33b38b0fcff3b3719615bbd |
| id | 2f91193c-c462-4bcf-ad87-692c3d263cfd |
| image | cirros-0.3.4-x86_64-uec (7bdff9a3-d051-4e75-bcd3-de69dbffe063) |
| key_name | - |
| metadata | {} |
| name | test_rebuild |
| os-extended-volumes:volumes_attached | [{"id": "daaddb77-4257-4ccd-86f2-220b31a0ce9b"}] |
| progress | 0 |
| public network | 2001:db8::19, 172.24.4.24 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | b8112a8d8227490eba99419b8a8c2555 |
| updated | 2015-08-06T04:10:00Z |
| user_id | ed64bccd0227444fa02dbd7695769a7d |
+--------------------------------------+----------------------------------------------------------------+

add using virsh to see the xml, we can see that the instance is still booted from the volume

</disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.250.10.17:3260-iscsi-iqn.2010-10.org.openstack:volume-daaddb77-4257-4ccd-86f2-220b31a0ce9b-lun-1'/>
      <target dev='vda' bus='virtio'/>
      <serial>daaddb77-4257-4ccd-86f2-220b31a0ce9b</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>

Changed in nova:
assignee: nobody → Zhenyu Zheng (zhengzhenyu)
tags: added: libvirt rebuild volumes
tags: removed: rebuild
Kashyap Chamarthy (kashyapc) wrote :

Hi Zhenyu,

I'm afraid I could not parse your problem description correctly, so I don't know what exactly is the problem you're facing.

Would be useful if you can write a list of sequential steps to reproduce the problem you're facing.

[A friendly note: https://wiki.openstack.org/wiki/BugFilingRecommendations]

Changed in nova:
status: New → Incomplete

Fix proposed to branch: master
Review: https://review.openstack.org/305079

Changed in nova:
assignee: Zhenyu Zheng (zhengzhenyu) → Laszlo Hegedus (laszlo-hegedus)
status: Incomplete → In Progress
Changed in nova:
assignee: Laszlo Hegedus (laszlo-hegedus) → Zhenyu Zheng (zhengzhenyu)
Changed in nova:
assignee: Zhenyu Zheng (zhengzhenyu) → Laszlo Hegedus (laszlo-hegedus)

Change abandoned by Laszlo Hegedus (<email address hidden>) on branch: master
Review: https://review.openstack.org/305079
Reason: I abandon this, since a proper solution would require rethinking of rebuild. It is not defined yet how rebuild should work with volume-backed instances. Someone should write a bp about this.

Changed in nova:
assignee: Laszlo Hegedus (laszlo-hegedus) → nobody
status: In Progress → Confirmed

Hi
I upgrade system to liberty but problem still remain
So for rebuild process with ceph backend i must do these step
- Delete instance
- Create neutron port with ip same as before
- Create new volumes base from os image template with size same as before
- Create new instance with same as flavor before, new volume and neutron port created before.

Fix proposed to branch: master
Review: https://review.openstack.org/442295

Changed in nova:
assignee: nobody → Akira KAMIO (kamio)
status: Confirmed → In Progress

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/442295
Reason: Abandoning as a duplicate of https://review.openstack.org/#/c/305079/

Matt Riedemann (mriedem) on 2017-10-03
tags: added: rebuild

Reviewed: https://review.openstack.org/509282
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ae9542eb7da22ad6bcd59145c56b2781aa7b507a
Submitter: Jenkins
Branch: master

commit ae9542eb7da22ad6bcd59145c56b2781aa7b507a
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 3 16:34:26 2017 -0400

    api-ref: add note about rebuild not replacing volume-backed root disk

    We've had several bugs about this over the years, and until we
    actually decide to fix it (or not), we should point out the known
    limitation that volume-backed root disks are not replaced during
    a rebuild.

    Like, if you have a volume-backed instance and rebuild with a new
    image, the root disk is still the volume with the original image.

    Change-Id: I145cab88f782e4b1e630cc432322bc8436413e71
    Related-Bug: #1482040

Matt Riedemann (mriedem) on 2017-10-06
Changed in nova:
status: In Progress → Confirmed
assignee: Akira KAMIO (kamio) → nobody
summary: - rebuild volume-backed instance with image
+ rebuild volume-backed instance with image does not replace image in root
+ disk
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers