RBD backed instance can't shutdown and restart
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Undecided
|
Mike Perez | ||
Havana |
Fix Released
|
Undecided
|
Unassigned | ||
OpenStack Dashboard (Horizon) |
Invalid
|
Undecided
|
Unassigned | ||
Ubuntu |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
Version: Havana w/ Ubuntu Repos. with Ceph for RBD.
When creating Launching a instance with "Boot from image (Creates a new volume)" this creates the instance fine and all is well however if you shutdown the instance I can't turn it back on again.
I get the following error in the nova-compute.log when trying to power on an shutdown instance.
#######
2013-10-29 00:48:33.859 2746 WARNING nova.compute.utils [req-89bbd72f-
2013-10-29 00:48:34.040 2746 WARNING nova.virt.
2013-10-29 00:48:34.578 2746 ERROR nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.
#######
On Closer inspection It seems the libvirt.xml file for the instance gets all screwed up.
This is the libvirt.xml file for this instance before shutdown.
#######
<domain type="kvm">
<uuid>
<name>
<memory>
<vcpu>1</vcpu>
<sysinfo type="smbios">
<system>
<entry name="manufactu
<entry name="product"
<entry name="version"
<entry name="serial"
<entry name="uuid"
</system>
</sysinfo>
<os>
<type>
<boot dev="hd"/>
<smbios mode="sysinfo"/>
</os>
<features>
<acpi/>
<apic/>
</features>
<clock offset="utc">
<timer name="pit" tickpolicy=
<timer name="rtc" tickpolicy=
</clock>
<cpu mode="host-model" match="exact"/>
<devices>
<disk type="network" device="disk">
<driver name="qemu" type="raw" cache="none"/>
<source protocol="rbd" name="volumes/
<host name="10.100.96.10" port="6789"/>
<host name="10.100.96.11" port="6789"/>
<host name="10.100.96.12" port="6789"/>
</source>
<auth username="volumes">
<secret type="ceph" uuid="13a673af-
</auth>
<target bus="virtio" dev="vda"/>
<
</disk>
<interface type="bridge">
<mac address=
<model type="virtio"/>
<source bridge=
<target dev="tap125aa65
</interface>
<serial type="file">
<source path="/
</serial>
<serial type="pty"/>
<input type="tablet" bus="usb"/>
<graphics type="vnc" autoport="yes" keymap="en-us" listen=
</devices>
</domain>
#######
All looks fine here i can see the ceph mons etc.
Next i'll shutdown the instance and as expected the file stays the same now this is the file if i try to power the instance back on again.
you'll notice all of the details regarding the disk have changed and now it thinks its a qcow2 disk located on the local hard drive... how does this happen?
#######
<domain type="kvm">
<uuid>
<name>
<memory>
<vcpu>1</vcpu>
<sysinfo type="smbios">
<system>
<entry name="manufactu
<entry name="product"
<entry name="version"
<entry name="serial"
<entry name="uuid"
</system>
</sysinfo>
<os>
<type>
<boot dev="hd"/>
<smbios mode="sysinfo"/>
</os>
<features>
<acpi/>
<apic/>
</features>
<clock offset="utc">
<timer name="pit" tickpolicy=
<timer name="rtc" tickpolicy=
</clock>
<cpu mode="host-model" match="exact"/>
<devices>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2" cache="none"/>
<source file="/
<target bus="virtio" dev="vda"/>
</disk>
<interface type="bridge">
<mac address=
<model type="virtio"/>
<source bridge=
<target dev="tap125aa65
</interface>
<serial type="file">
<source path="/
</serial>
<serial type="pty"/>
<input type="tablet" bus="usb"/>
<graphics type="vnc" autoport="yes" keymap="en-us" listen=
</devices>
</domain>
#######
Changed in nova: | |
status: | New → Confirmed |
status: | Confirmed → In Progress |
assignee: | nobody → Mike Perez (thingee) |
tags: | added: havana-backport-potential |
Changed in nova: | |
milestone: | none → icehouse-1 |
Changed in nova: | |
status: | Fix Committed → Fix Released |
Changed in nova: | |
milestone: | icehouse-1 → 2014.1 |
I've found if i create the instance via horizon using the "Launch Instance"
Using Instance Boot Source: Boot from image (creates a new volume)
Selecting Image etc,
when powered off and back on again the error happens every time.
However!
If I create the volume first and use a "Boot from Volume" Option and selecting the volume i precreated with the image,
the Instance it seems it can power off and on without any issues.
So could be a horizon issue in how its storing the data in sql?