Config Drive not created when boot from volume

[Tao Zhou]

I am trying to launch my instance from cinder volume with heat template, but no success.
My template will first create a volume from a glance image and then boot from this volume.
It seems that libvirt is looking for /var/lib/nova/instances/<instance-id>/disk, while my instance is booted from volume, the <instance-id> directory does not even exist.

In scheduler.log, I see:
Error from last host: compute-08 (node compute-08.local): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1305, in _build_instance\n set_access_ip=set_access_ip)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 393, in decorated_function\n return function(self, context, *args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1717, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1714, in _spawn\n block_device_info)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2265, in spawn\n block_device_info)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3656, in _create_domain_and_network\n power_on=power_on)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3559, in _create_domain\n domain.XMLDesc(0))\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3554, in _create_domain\n domain.createWithFlags(launch_flags)\n', u' File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 179, in doit\n result = proxy_call(self._autowrap, f, *args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 139, in proxy_call\n rv = execute(f,*args,**kwargs)\n', u' File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 77, in tworker\n rv = meth(*args,**kwargs)\n', u' File "/usr/lib64/python2.6/site-packages/libvirt.py", line 708, in createWithFlags\n if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', u"libvirtError: cannot open file '/var/lib/nova/instances/3310bf49-82d0-467d-8b9b-de8d453cdef8/disk': No such file or directory\n"]

[Sean Dague (sdague)]

Please provide heat templates as well as the nova config, I expect part of the issue is the options that heat is passing to Nova.

[Launchpad Janitor (janitor)]

[Expired for heat because there has been no activity for 60 days.]

[Sergey Motovilovets (motovilovets-sergey)]

Hey

I'm facing exactly the same problem with Icehouse's Heat and Ceph as a backend for Cinder and Glance

Here's a part of my Heat template:

  volume_smth:
    type: OS::Cinder::Volume
    properties:
      size: { get_param: root_part_size }
      image: { get_param: image }

  port_smth:
    type: OS::Neutron::Port
    properties:
      network_id: { get_resource: private_net }
      security_groups: [ { get_resource: security_group } ]
      fixed_ips:
        - subnet_id: { get_resource: private_subnet }
          ip_address:
            str_replace:
              template: $net.10
              params:
                $net: { get_param: private_net_part }

  server_smth:
    type: OS::Nova::Server
    properties:
      name:
        str_replace:
          template: $dc_name-smth
          params:
            $dc_name: { get_param: dc_name }
      flavor: { get_param: flavor }
      block_device_mapping:
        - device_name: vda
          volume_id: { get_resource: volume_smth }
          delete_on_termination: true
      networks:
        - port: { get_resource: port_smth }

# heat stack-create -f stack.yml -e env.yml stack

# cinder list | grep stack
...
9ada74f7-8260-4731-86f9-b8b64b62cf1f | available | stack-volume_smth-cxx4gc4oigpz
...

# rbd -p volumes ls | grep 9ada
volume-9ada74f7-8260-4731-86f9-b8b64b62cf1f

And here's what reaches the compute node:

    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="none"/>
      <source file="/var/lib/nova/instances/7cb81b39-1cb9-42e8-95d5-f377ea7b9039/disk"/>
      <target bus="virtio" dev="vda"/>
    </disk>

Causing

libvirtError: Failed to open file '/var/lib/nova/instances/7cb81b39-1cb9-42e8-95d5-f377ea7b9039/disk'

Same thing happens when I try to boot from volume manually, issuing:
# nova boot --flavor m1.medium --block-device source=volume,id=9ada74f7-8260-4731-86f9-b8b64b62cf1f,shutdown=preserve,bootindex=0 --nic net-id=cee71b20-efec-46b2-bfe6-2cf0976e7271 testing

# nova list | grep testing
testing | ERROR | - | NOSTATE

But it works like a charm with an extra option dest=volume (found this hint in https://bugs.launchpad.net/mos/+bug/1391950) :
#nova boot --flavor m1.medium --block-device source=volume,id=9ada74f7-8260-4731-86f9-b8b64b62cf1f,shutdown=preserve,dest=volume,bootindex=0 --nic net-id=cee71b20-efec-46b2-bfe6-2cf0976e7271 testing

# nova list | grep testing
testing | ACTIVE

in this case, compute node finally gets
      <driver name="qemu" type="raw" cache="none"/>
      <source protocol="rbd" name="volumes/volume-9ada74f7-8260-4731-86f9-b8b64b62cf1f">

[Steve Lipinski (steve-lipinski)]

I'm encountering the same issue. Trying to use a Cinder Volume as the boot device/volume for a VM.

Here is the Heat template:

heat_template_version: '2013-05-23'
resources:
  myip:
    properties:
      name: myip
      network_id: 97304220-96ff-41d4-9009-82c68ba0f47b
      security_groups: [ default ]
    type: OS::Neutron::Port
  myserver:
    properties:
      availability_zone: zone0
      block_device_mapping:
      - delete_on_termination: 'true'
        device_name: vda
        volume_id: {get_resource: 'myserver:vda'}
      config_drive: 'True'
      flavor: 4x8_50
      name: myserver
      networks:
      - port: {get_resource: 'myip'}
    type: OS::Nova::Server
  myserver:vda:
    properties:
      availability_zone: nova
      image: myimage
      name: 'myserver:vda'
      size: '150'
    type: OS::Cinder::Volume

I do not have access to the compute node for logs, but here is what is reports by nova for this instance:
| fault | {"message": "No valid host was found. Exceeded max scheduling attempts 3 for instance 0603db4e-afc9-4899-908b-e41320b6780c", "code": 500, "created": "2015-01-30T00:40:43Z"} |

[Deliang Fan (vanderliang)]

@Sergey Motovilovets (motovilovets-sergey)

Your are right, it hits https://bugs.launchpad.net/mos/+bug/1391950, we need to add an extra option dest=volume when boot instance from volume in OS::Nova::Server

[Deliang Fan (vanderliang)]

We need to add another property named block_device_mapping_v2 which contains the attribution destination_type

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/153454

[Christoph Dwertmann (cdwertmann)]

I can confirm that the patch submitted is working for me. A glance image is cloned into a cinder volume, which is then used to boot an instance (no need to use "device_name: vda").

 I'm using this heat template:

heat_template_version: 2013-05-23

description: Simple template to deploy a single compute instance

resources:
  my_instance:
    type: OS::Nova::Server
    properties:
      key_name: c
      flavor: m1.tiny
      networks:
        - network: cdw_network
      block_device_mapping_v2:
        - volume_id: { get_resource: server_volume_01 }
          delete_on_termination: true

  server_volume_01:
    type: OS::Cinder::Volume
    properties:
      image: Ubuntu 14.10
      volume_type: tier1
      size: 3

[Steve Lipinski (steve-lipinski)]

Can this be backported to IceHouse and Juno?

[Steve Baker (steve-stevebaker)]

I don't think a backport would help, since it wouldn't work unless the as-yet unreleased python-novaclient 0.21.0 ended up being packaged in the icehouse and juno downstream repositories too

[Christoph Dwertmann (cdwertmann)]

The patch actually works fine on my icehouse installation. I did not upgrade python-novaclient or any other packages to releases above icehouse.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/153454
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=0dfd2f9ba0e6b90d93a914b90fe2cf2dfd034b39
Submitter: Jenkins
Branch: master

commit 0dfd2f9ba0e6b90d93a914b90fe2cf2dfd034b39
Author: fandeliang <email address hidden>
Date: Fri Feb 6 06:42:59 2015 -0500

    Fix config drive not created when boot from volume

    When booting a instance from volume, destination_type is needed
    to set to volume in block device mapping v2. However, parameter
    block device mapping v2 is not in heat when booting an instance,
    so we need to add it as property for OS::Nova::Server.

    Change-Id: Ie56358ed6fdb7cda4a28d546817c31e71ca179b3
    Closes-Bug: #1358624