nova boot fails with rbd backend

[Jens Rosenboom]

Booting a VM in a plain devstack setup with ceph enabled, I get an error like:

libvirtError: internal error: process exited while connecting to monitor: qemu-system-x86_64: -drive file=rbd:vmz/27dcd57f-948f-410c-830f-48d8fda0d968_disk.config:id=cindy:key=AQA00PxTiFa0MBAAQ9Uq9IVtBwl/pD8Fd9MWZw==:auth_supported=cephx\;none:mon_host=192.168.122.76\:6789,if=none,id=drive-ide0-1-1,readonly=on,format=raw,cache=writeback: error reading header from 27dcd57f-948f-410c-830f-48d8fda0d968_disk.config

even though config_drive is set to false.

This seems to be related to https://review.openstack.org/#/c/112014/, if I revert ecce888c469c62374a3cc43e3cede11d8aa1e799 everything works fine.

[Jens Rosenboom (frickler)]

After looking around a bit further, I'm not sure whether the correct fix would be to revert using image_type=None instead of 'raw' again at https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3315 or whether this just uncovers some other bug that has been hiding out for longer.

In the log before the error, the config drive is created on local storage, but why is it created at all if the instance does not call for it?

[Mehdi Abaakouk (sileht)]

https://review.openstack.org/#/c/116847/ should be fix this issue

[Sébastien Han (sebastien-han)]

Can't you simply disable the config drive? If you can't then you're probably hitting another bug.
Are you be any chance using nova-network because I remember that I hit a bug like this a couple of months ago.

[Jens Rosenboom (frickler)]

As discussed in irc, devstack sets "force_config_drive=always" in nova.conf. If I disable this, I don't get this bug, so that goes through as a workaround. Still the experience for new users doing their first testing with devstack+ceph will be: Its broken.

With the patch from https://review.openstack.org/#/c/116847/ I get other errors like:

ProcessExecutionError: Unexpected error while running command.
Command: rbd import --pool vmz /opt/stack/data/nova/instances/_base/disk.config 4c76d8a4-
f927-40ef-9106-8f6477a73819_disk.config --new-format --id cindy --conf /etc/ceph/ceph.conf
Stderr: 'rbd: error opening /opt/stack/data/nova/instances/_base/disk.config\nrbd: import
 failed: (2) No such file or directory\n'

Looks like now nova tries to create the config drive in rbd, but fails because of the badly quoted whitespace in the filename.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/116847
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0b01e846d40f3b343da9ebe1dae89cca8bc2ac66
Submitter: Jenkins
Branch: master

commit 0b01e846d40f3b343da9ebe1dae89cca8bc2ac66
Author: Mehdi Abaakouk <email address hidden>
Date: Tue Aug 26 12:29:30 2014 +0200

    libvirt: Uses correct imagebackend for configdrive

    When the configdrive file is created it must be moved to the
    configured imagebackend, not always the raw one.

    Otherwise nova can't boot an instance with a configdrive attached
    when the rbd backend is configured.

    Related to bug #1246201
    Closes-bug: #1361840

    Change-Id: I59578d9b87a36ff4417a30ff14c5774c54698cf3

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/117292
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=228d0221763b12f11ecbacde4db38b1151f96e31
Submitter: Jenkins
Branch: master

commit 228d0221763b12f11ecbacde4db38b1151f96e31
Author: Mehdi Abaakouk <email address hidden>
Date: Wed Aug 27 17:07:43 2014 +0200

    libvirt: reworks configdrive creation

    This refactor the creation of the configdrive by using
    the same code schema as local/ephemeral/swap/disk.

    Now this is the imagebackend that creates the configdrive like any other
    disk attached to VM. This ensures that the configdrive file is created
    where the imagebackend expect it.

    This also removes the assumption that the configdrive was always created
    at the right place, that was not true for rbd and lvm.

    Related-Bug: #1361840

    Change-Id: Ia03a83d593794c8b2a644360790cb15de9dea6e9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/123073

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/159087

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mehdi Abaakouk (<email address hidden>) on branch: master
Review: https://review.openstack.org/159087
Reason: Merged in previous review

[OpenStack Infra (hudson-openstack)]

Change abandoned by Joe Gordon (<email address hidden>) on branch: master
Review: https://review.openstack.org/123073
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/123073
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=adecf780d3ed4315e4ce305cb1821d493650494b
Submitter: Jenkins
Branch: master

commit adecf780d3ed4315e4ce305cb1821d493650494b
Author: Michael Still <email address hidden>
Date: Tue Nov 25 15:42:47 2014 +0300

    Handle config drives being stored on rbd

    rbd is the only example of a currently supported image storage
    backend where it makes sense to put the config drive in the
    configured storage backend instead of local hypervisor disk. I
    don't think this makes sense for LVM, where we would be creating
    a LV for a tens of megabytes file, which seems like overkill to
    me. The other storage backends use local disk for their data
    already.

    This use case was covered by the now reverted changes:
      228d0221763b12f11ecbacde4db38b1151f96e31
      0b01e846d40f3b343da9ebe1dae89cca8bc2ac66
      ecce888c469c62374a3cc43e3cede11d8aa1e799

    Support this special case by moving the image to rbd once it has
    been created in the local instance directory on the hypervisor.

    I've tested this change in devstack and it works.

    Related-bug: #1369627
    Related-bug: #1361840
    Related-bug: #1246201

    Co-Authored-By: Mehdi Abaakouk <email address hidden>
    Co-Authored-By: Dan Smith <email address hidden>

    Change-Id: Ia3ca5a18c79d62b71b9c042a612d12dd074b245e

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/214773

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rosario Di Somma (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/214773