VM won't start if cloud-init is enabled

[JB Broccard]

Hi,

I am trying to leverage cloud-init on my kolla / openstack system. The flat_injected and force_config_drive are set to True in my nova.conf file (that I've changed in /etc/kolla/config/... directory before deploying everything). I have created the /usr/lib/python2.7/site-packages/nova/virt/interfaces.template file which is the default location in Kolla / Kilo.

Then, I boot a VM with that command:
$ docker-ostk nova boot --flavor m1.medium --image testVMimage --nic net-id=<<id of my network>> --availability-zone nova:compute1 ext-vm2

The web UI reports this on the VM overview page: Build of instance <<my vm id>> was re-scheduled: [Errno 2] No such file or directory

The nova_compute container reports this log:
2016-01-26 20:40:21.631 1 INFO nova.virt.disk.vfs.api [req-c88b36ae-08fc-4f5b-8b0c-d6c9837bdb09 d8c27ead14bd4924a74fa42d160fbff3 140f3a63f0e04a84ad8359a64aec6c7d - - -] Unable to import guestfs, falling back to VFSLocalFS
2016-01-26 20:40:22.223 1 INFO nova.virt.libvirt.driver [req-c88b36ae-08fc-4f5b-8b0c-d6c9837bdb09 d8c27ead14bd4924a74fa42d160fbff3 140f3a63f0e04a84ad8359a64aec6c7d - - -] [instance: 7570c59a-ca42-4259-ba32-756cba2bf09b] Using config drive
2016-01-26 20:40:22.385 1 INFO nova.virt.libvirt.driver [req-c88b36ae-08fc-4f5b-8b0c-d6c9837bdb09 d8c27ead14bd4924a74fa42d160fbff3 140f3a63f0e04a84ad8359a64aec6c7d - - -] [instance: 7570c59a-ca42-4259-ba32-756cba2bf09b] Creating config drive at /var/lib/nova/instances/7570c59a-ca42-4259-ba32-756cba2bf09b/disk.config
2016-01-26 20:40:22.581 1 ERROR nova.compute.manager [req-c88b36ae-08fc-4f5b-8b0c-d6c9837bdb09 d8c27ead14bd4924a74fa42d160fbff3 140f3a63f0e04a84ad8359a64aec6c7d - - -] [instance: 7570c59a-ca42-4259-ba32-756cba2bf09b] Instance failed to spawn
2016-01-26 20:40:22.581 1 TRACE nova.compute.manager [instance: 7570c59a-ca42-4259-ba32-756cba2bf09b] Traceback (most recent call last):
2016-01-26 20:40:22.581 1 TRACE nova.compute.manager [instance: 7570c59a-ca42-4259-ba32-756cba2bf09b] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2473, in _build_resources
2016-01-26 20:40:22.581 1 TRACE nova.compute.manager [instance: 7570c59a-ca42-4259-ba32-756cba2bf09b] yield resources

Meanwhile, the compute hosts reports issue with NBD:
^C[root@blade6 ~]# ps -ef|grep defun
root 23263 22431 0 12:40 ? 00:00:00 [qemu-nbd] <defunct>
root 23264 22431 0 12:40 ? 00:00:00 [qemu-nbd] <defunct>
root 23344 4881 0 12:41 pts/0 00:00:00 grep --color=auto defun
[root@blade6 ~]# dmesg | grep -i nbd
[ 1065.822948] block nbd11: NBD_DISCONNECT
[ 1065.823140] block nbd11: Unexpected reply (ffff881fb9381c88)
[ 1065.823344] block nbd11: queue cleared
[ 1911.369869] block nbd4: NBD_DISCONNECT
[ 1911.369967] block nbd4: Unexpected reply (ffff883f830a9c88)
[ 1911.370189] block nbd4: queue cleared
[ 2008.686886] block nbd8: NBD_DISCONNECT

Not sure it's relevant though.

Question: is even cloud-init supported in Kolla? If so, what am I doing wrong?

Thanks,
JB

[JB Broccard (jbbroccard)]

the container was lacking geniso package. After adding it, it works.

[Sam Yaple (s8m)]

JB Broccard: Do you have a patch submitted upstream for this? This seems like a good candidate for you to submit a patch for

[JB Broccard (jbbroccard)]

Sam Yaple: this sounds more like a feature gap than a bug to me. Let me know what you think.

[sean mooney (sean-k-mooney)]

i stumbled across this issue last weekend also.

i have not so far been able to use cloud-init without a configdrive with kolla.

the nova-compute container is indeed missing the genios package to enable it to deploy with a config drive.
on ubuntu this is provided by genisoimage.

too me this feels more like a missing dependency rather then a new feature.

if you wanted to enable could-init with kolla as a feature i think it should also be enabled/document when using the nova metadata api and neutron metadata proxy.

if the scope of the patch is simply adding the missing dependency to enable the config drive to be built i think
it qualifies as a bug fix. the fact that the neutron metadata agent is deployed by kolla suggest that cloud-init should work but that is just my 2 cents

[JB Broccard (jbbroccard)]

Fair enough, Sean. I'm not used to discuss bug vs. feature gap, so i leave to you.
In my case, I had to follow these steps to work around the issue:
1. docker copy from host to container libusa* and geniso* packages
2. install the packages inside the container
That's all. Then, of course I had to modify the config file:
/etc/kolla/config/.../nova-compute.conf with
flat_injected = True
force_config_drive = True
Note: the "True" word is case sensitive otherwise the service/container won't start (I checked the code, it expects "True").

[sean mooney (sean-k-mooney)]

you can actully reporduce this issue without any config changes.

if you launch a vm from horizon in the advanced options just tick the Config drive box

after installing genisoimage in the nova_compute container the vm will boot fine.

well sam or the other cores really should make the call of feature vs bug but that was
just my interpreation based on the fact that the metadata service is deployed currently which suggest that at
some point there was the intention to supprot cloud init with kolla.

[Steven Dake (sdake)]

this is a bug - if nova implements it and horizon exposes it and its as easy as installing a package, we might as well fix the problem.

[sean mooney (sean-k-mooney)]

@jbbroccard
do you want to fix this one?

im in meeting most of the day tomorrow but ill try and submit a patch
tomorow if i dont see one form you.

@sdake
The configuration where a config driver is used to store the cloud init files can be eaily fixed by
installing the genisoimage package however i am not sure how to fix the configuration where the cloud
init files are server by the nova api via the neutron metatda agent. as far as i can tell kolla is
deploying the agent but the cloud init queries to 169.254.169.254 are not being correctly
proxied to the nova api. fixing this for the config drive case though would at least enable cloud init to function on a
kolla deploy ment and enabling the metadata method could done as a follow up patch once we
figure out why that is not working.

[sean mooney (sean-k-mooney)]

review: https://review.openstack.org/285598

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/285598
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=768c7d706125e118774bc01c30db5b5b765e080c
Submitter: Jenkins
Branch: master

commit 768c7d706125e118774bc01c30db5b5b765e080c
Author: Sean Mooney <email address hidden>
Date: Fri Feb 26 21:50:00 2016 +0000

    Adds genisoimage package dependency to nova-compute

    - This change enables the nova-compute agent to generate
      config drives to attach the an instance.
    - This allows the config drive to be used to pass
      cloud-init config files to a guest enableing first boot
      configuration.

    Change-Id: Id17741b3fce1382350914c1b6c818b0020fe839a
    Closes-Bug: #1538288

[sean mooney (sean-k-mooney)]

cherry-picked to stable/mitaka for review

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/297637
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=6dcb4e23d5bd7e3d8bbc2c3c49f82f24f21a51d4
Submitter: Jenkins
Branch: stable/mitaka

commit 6dcb4e23d5bd7e3d8bbc2c3c49f82f24f21a51d4
Author: Sean Mooney <email address hidden>
Date: Fri Feb 26 21:50:00 2016 +0000

    Adds genisoimage package dependency to nova-compute

    - This change enables the nova-compute agent to generate
      config drives to attach the an instance.
    - This allows the config drive to be used to pass
      cloud-init config files to a guest enabling first boot
      configuration.

    Change-Id: Id17741b3fce1382350914c1b6c818b0020fe839a
    Closes-Bug: #1538288
    (cherry picked from commit 768c7d706125e118774bc01c30db5b5b765e080c)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/kolla 2.0.0 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/kolla 1.1.0 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/kolla 3.0.0.0b1 development milestone.