RBD backed instance can't shutdown and restart

[Quenten Grasso]

Version: Havana w/ Ubuntu Repos. with Ceph for RBD.

When creating Launching a instance with "Boot from image (Creates a new volume)" this creates the instance fine and all is well however if you shutdown the instance I can't turn it back on again.

I get the following error in the nova-compute.log when trying to power on an shutdown instance.

#######################################################################################
2013-10-29 00:48:33.859 2746 WARNING nova.compute.utils [req-89bbd72f-2280-4fac-802a-1211ec774980 27106b78ceac4e389558566857a7875f 464099f86eb94d049ed1f7b0f0144275] [instance: cc370f6d-4be0-4cd3-9f20-bf86f5ad7c09] Can't access image $
2013-10-29 00:48:34.040 2746 WARNING nova.virt.libvirt.vif [req-89bbd72f-2280-4fac-802a-1211ec774980 27106b78ceac4e389558566857a7875f 464099f86eb94d049ed1f7b0f0144275] Deprecated: The LibvirtHybridOVSBridgeDriver VIF driver is now de$
2013-10-29 00:48:34.578 2746 ERROR nova.openstack.common.rpc.amqp [req-89bbd72f-2280-4fac-802a-1211ec774980 27106b78ceac4e389558566857a7875f 464099f86eb94d049ed1f7b0f0144275] Exception during message handling
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp **args)
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp payload)
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in decorated_function
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp pass
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in decorated_function
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in decorated_function
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1832, in start_instance
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp self._power_on(context, instance)
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1819, in _power_on
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp block_device_info)
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1948, in power_on
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp self._hard_reboot(context, instance, network_info, block_device_info)
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1903, in _hard_reboot
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp block_device_info)
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4318, in get_instance_disk_info
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp dk_size = int(os.path.getsize(path))
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/genericpath.py", line 49, in getsize
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp return os.stat(filename).st_size
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp OSError: [Errno 2] No such file or directory: '/var/lib/nova/instances/cc370f6d-4be0-4cd3-9f20-bf86f5ad7c09/disk'
2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp
#######################################################################################

On Closer inspection It seems the libvirt.xml file for the instance gets all screwed up.

This is the libvirt.xml file for this instance before shutdown.

#######################################################################################
<domain type="kvm">
  <uuid>dc9749cd-0002-41c9-ac55-5f691637146a</uuid>
  <name>instance-00000004</name>
  <memory>524288</memory>
  <vcpu>1</vcpu>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">OpenStack Foundation</entry>
      <entry name="product">OpenStack Nova</entry>
      <entry name="version">2013.2</entry>
      <entry name="serial">4c4c4544-0053-3210-8032-b6c04f5a5931</entry>
      <entry name="uuid">dc9749cd-0002-41c9-ac55-5f691637146a</entry>
    </system>
  </sysinfo>
  <os>
    <type>hvm</type>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset="utc">
    <timer name="pit" tickpolicy="delay"/>
    <timer name="rtc" tickpolicy="catchup"/>
  </clock>
  <cpu mode="host-model" match="exact"/>
  <devices>
    <disk type="network" device="disk">
      <driver name="qemu" type="raw" cache="none"/>
      <source protocol="rbd" name="volumes/volume-5abadeb8-49e9-4628-a54d-d742d6f2e012">
        <host name="10.100.96.10" port="6789"/>
        <host name="10.100.96.11" port="6789"/>
        <host name="10.100.96.12" port="6789"/>
      </source>
      <auth username="volumes">
        <secret type="ceph" uuid="13a673af-ff80-3036-8310-4c72f566673d"/>
      </auth>
      <target bus="virtio" dev="vda"/>
      <serial>5abadeb8-49e9-4628-a54d-d742d6f2e012</serial>
    </disk>
    <interface type="bridge">
      <mac address="fa:16:3e:45:17:d4"/>
      <model type="virtio"/>
      <source bridge="qbr125aa659-01"/>
      <target dev="tap125aa659-01"/>
    </interface>
    <serial type="file">
      <source path="/var/lib/nova/instances/dc9749cd-0002-41c9-ac55-5f691637146a/console.log"/>
    </serial>
    <serial type="pty"/>
    <input type="tablet" bus="usb"/>
    <graphics type="vnc" autoport="yes" keymap="en-us" listen="10.100.32.10"/>
  </devices>
</domain>
#######################################################################################

All looks fine here i can see the ceph mons etc.

Next i'll shutdown the instance and as expected the file stays the same now this is the file if i try to power the instance back on again.

you'll notice all of the details regarding the disk have changed and now it thinks its a qcow2 disk located on the local hard drive... how does this happen?

#######################################################################################

<domain type="kvm">
  <uuid>dc9749cd-0002-41c9-ac55-5f691637146a</uuid>
  <name>instance-00000004</name>
  <memory>524288</memory>
  <vcpu>1</vcpu>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">OpenStack Foundation</entry>
      <entry name="product">OpenStack Nova</entry>
      <entry name="version">2013.2</entry>
      <entry name="serial">4c4c4544-0053-3210-8032-b6c04f5a5931</entry>
      <entry name="uuid">dc9749cd-0002-41c9-ac55-5f691637146a</entry>
    </system>
  </sysinfo>
  <os>
    <type>hvm</type>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset="utc">
    <timer name="pit" tickpolicy="delay"/>
    <timer name="rtc" tickpolicy="catchup"/>
  </clock>
  <cpu mode="host-model" match="exact"/>
  <devices>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="none"/>
      <source file="/var/lib/nova/instances/dc9749cd-0002-41c9-ac55-5f691637146a/disk"/>
      <target bus="virtio" dev="vda"/>
    </disk>
    <interface type="bridge">
      <mac address="fa:16:3e:45:17:d4"/>
      <model type="virtio"/>
      <source bridge="qbr125aa659-01"/>
      <target dev="tap125aa659-01"/>
    </interface>
    <serial type="file">
      <source path="/var/lib/nova/instances/dc9749cd-0002-41c9-ac55-5f691637146a/console.log"/>
    </serial>
    <serial type="pty"/>
    <input type="tablet" bus="usb"/>
    <graphics type="vnc" autoport="yes" keymap="en-us" listen="10.100.32.10"/>
  </devices>
</domain>

#######################################################################################

[Quenten Grasso (qgrasso-d)]

I've found if i create the instance via horizon using the "Launch Instance"

Using Instance Boot Source: Boot from image (creates a new volume)

Selecting Image etc,

when powered off and back on again the error happens every time.

However!

If I create the volume first and use a "Boot from Volume" Option and selecting the volume i precreated with the image,
the Instance it seems it can power off and on without any issues.

So could be a horizon issue in how its storing the data in sql?

[Josh Durgin (jdurgin)]

The 'Boot from image (creates a new volume)' code path uses the new block_device_mapping_v2 api. This is most likely a bug in the implementation of this in nova that affects all volumes backends, not just rbd.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Blane Bramble (blane)]

This seems to be similar to Bug #1248695

[Blane Bramble (blane)]

I believe this is a problem with the source_type being set to image and the destination_type being set to volume which prevents the DriverVolumeBlockDevice._transform code from firing. A work around is attached that just detects the source_type as image and forces it to volume for the check - a better solution would be to not set it to image in the first place but I haven't tracked where this is done yet.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Work-around rather than fix" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Quenten Grasso (qgrasso-d)]

I Tried to implement the patch from Blane however doesn't seem to work.

When trying to launch a instance it now fails when it try's to map the block device.

Any ideas?

[Blane Bramble (blane)]

Hi Quenten, what version are you installing, and what error do you get?

[Quenten Grasso (qgrasso-d)]

Hi Blane,

Sorry for my late reply, I'm using the ubuntu Havana with the latest from "updates" repo.

Q

[Mike Perez (thingee)]

I'm also having the same problem. Using RBD for both Cinder and Glance image backend. If I create an instance and select in horizon 'boot from image (create new volume)', soft/hard reboot fails later.

I get the following error from the compute manager:

2013-11-19 09:56:26.036 ERROR nova.openstack.common.rpc.amqp [req-b5d86ef0-f03d-4708-b94a-e94f30055b18 admin demo] Exception during message handling
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp **args)
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 353, in decorated_function
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 90, in wrapped
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp payload)
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 73, in wrapped
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 243, in decorated_function
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp pass
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 229, in decorated_function
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 294, in decorated_function
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 271, in decorated_function
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 258, in decorated_function
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 2121, in reboot_instance
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amqp block_device_info = self._get_instance_volume_block_device_info(
2013-11-19 09:56:26.036 TRACE nova.openstack.common.rpc.amq...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/57406

[Nikola Đipanov (ndipanov)]

I've acked the fix on the master branch - we need this on stable/havana too. I can't seem to propose the bug, but will happily do the backport.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/57406
Committed: http://github.com/openstack/nova/commit/85d0ace169be513c30b09e13a35fa7d912f5b380
Submitter: Jenkins
Branch: master

commit 85d0ace169be513c30b09e13a35fa7d912f5b380
Author: Mike Perez <email address hidden>
Date: Wed Nov 20 01:22:38 2013 -0800

    Include image block device maps in info

    In cases for example soft/hard reboot, we were just getting volume and
    snapshot block device maps. For block devices with a source type image,
    Nova would fall back on a local disk location and fail.

    This change allows us to include block devices with a source of an
    image, so that we correctly get the right the location of the block
    device.

    Closes-Bug: #1245719
    Change-Id: I1f726293e85183d4b84c05b635a7c606a092992f

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/58102

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/58102
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=dfde60b4bb05c45daedc2eaa98fcd772e647fba5
Submitter: Jenkins
Branch: stable/havana

commit dfde60b4bb05c45daedc2eaa98fcd772e647fba5
Author: Mike Perez <email address hidden>
Date: Wed Nov 20 01:22:38 2013 -0800

    Include image block device maps in info

    In cases for example soft/hard reboot, we were just getting volume and
    snapshot block device maps. For block devices with a source type image,
    Nova would fall back on a local disk location and fail.

    This change allows us to include block devices with a source of an
    image, so that we correctly get the right the location of the block
    device.

    Closes-Bug: #1245719
    Change-Id: I1f726293e85183d4b84c05b635a7c606a092992f
    (cherry picked from commit 85d0ace169be513c30b09e13a35fa7d912f5b380)

[Akihiro Motoki (amotoki)]

Related to Horizon, does this bug still exist after nova patch is merged.
I don't have Ceph environment now and cannot check it.
Horizon team needs more information about the current status. Mark it "Incomplete".

[Gary W. Smith (gary-w-smith)]

Marking the horizon bug as Invalid due to lack of confirmation and low probability that there is a bug in horizon