NBD mount errors when booting an instance from volume

[Simon Pasquier]

My environment:
- Grizzly OpenStack (installed from Ubuntu repository)
- Network using Quantum
- Cinder backed up by a Ceph cluster

I'm able to boot an instance from a volume but it takes a long time for the instance to be active. I've got warnings in the logs of the nova-compute node (see attached file). The logs show that the problem is related to file injection in the disk image which isn't required/relevant when booting from a volume.

[Kashyap Chamarthy (kashyapc)]

Simon, can you provide some more descriptive detail on how you triggered it?

  - For instance, what's the nova CLI you used?

 Is it something like:

   $ nova boot --flavor 1 --image e1b71961-d66d-4315-8e83-32aa1bd44f3f \
   --block_device_mapping vda=2c370395-7f59-4c89-b312-ba35dbb986c0:::0 \
   --key_name oskey f17_volume_backed

Or something else?

It'll also be help triage things quicker, if you can provide as much verbose details as possible.

Some related reading -- https://wiki.openstack.org/wiki/BugFilingRecommendations

Thanks.

[Simon Pasquier (simon-pasquier)]

All nodes are running Ubuntu 13.04 (Quantal).

Here is the list of cinder, nova & ceph packages on the various nodes:
cinder-api (1:2013.1-0ubuntu2)
cinder-common (1:2013.1-0ubuntu2)
cinder-scheduler (1:2013.1-0ubuntu2)
nova-ajax-console-proxy (1:2013.1-0ubuntu2.1)
nova-api (1:2013.1-0ubuntu2.1)
nova-cert (1:2013.1-0ubuntu2.1)
nova-common (1:2013.1-0ubuntu2.1)
nova-compute (1:2013.1-0ubuntu2.1)
nova-compute-kvm (1:2013.1-0ubuntu2.1)
nova-conductor (1:2013.1-0ubuntu2.1)
nova-consoleauth (1:2013.1-0ubuntu2.1)
nova-doc (1:2013.1-0ubuntu2.1)
nova-network (1:2013.1-0ubuntu2.1)
nova-novncproxy (1:2013.1-0ubuntu2.1)
nova-scheduler (1:2013.1-0ubuntu2.1)
python-cinder (1:2013.1-0ubuntu2)
python-cinderclient (1:1.0.3-0ubuntu1)
python-nova (1:2013.1-0ubuntu2.1)
python-novaclient (1:2.13.0-0ubuntu1)
nova-compute-kvm (1:2013.1-0ubuntu2.1)
cinder-volume (1:2013.1-0ubuntu2)
ipxe-qemu (1.0.0+git-4.d6b0b76-0ubuntu2)
kvm-ipxe (1.0.0+git-4.d6b0b76-0ubuntu2)
qemu-keymaps (1.4.0+dfsg-1expubuntu4)
qemu-kvm (1.4.0+dfsg-1expubuntu4)
qemu-system-common (1.4.0+dfsg-1expubuntu4)
qemu-system-x86 (1.4.0+dfsg-1expubuntu4)
qemu-utils (1.4.0+dfsg-1expubuntu4)
ceph (0.56.4-1quantal)
ceph-common (0.56.6-1quantal)
ceph-fs-common (0.56.6-1quantal)
ceph-fuse (0.56.6-1quantal)
ceph-mds (0.56.4-1quantal)
libcephfs1 (0.56.6-1quantal)
python-ceph (0.56.6-1quantal)

Yes the instance was booted with the nova CLI as in your comment.

HTH, tell me if you need more stuff.

[Michael Davies (mrda)]

The problem here is not just speed of instance boot, but the file injection doesn't succeed.

This is non-trivial to resolve, and it's likely that file injection may just disappear in Icehouse (in preference to using cloud-init). There's support for this position here: http://<email address hidden>/msg04930.html

In addition, there may be multiple attached volumes at different mount points, and finding out which ones you need to mount and inject files into is non-trivial. While it is solvable, it will be complex code, and it's probably too late for this amount of new functionality to land in Havanna at this point in time due to its complexity.

I propose a simpler fix to skip the file injection attempt for booted volumes for now which will speed up instance creation (but WARNing the operator), and addressing this further in Icehouse.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/48533

[Michael Davies (mrda)]

Fix added in https://review.openstack.org/48533

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/48533
Committed: http://github.com/openstack/nova/commit/f59c2d64ac8f0523c587e23b2d1c59fdfe7ea131
Submitter: Jenkins
Branch: master

commit f59c2d64ac8f0523c587e23b2d1c59fdfe7ea131
Author: Michael Davies <email address hidden>
Date: Thu Oct 17 04:41:09 2013 -0700

    Fail quickly if file injection for boot volume

    Previously if the user attempted file injection on a booted volume it
    would fail and retry several times, which slowed down the instance
    spawning. Instead of attempting file injection, fail quickly, and
    log a warning

    Fix bug 1188543

    Change-Id: I264540de736c2bcb92567826fe5ba672e1244ba2

[Michael Davies (mrda)]

Sorry, this fix isn't released, but committed. I'll try and get the bug state reverted to the correct state.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/54460

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/54472

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/54472
Committed: http://github.com/openstack/nova/commit/119703b2742486e874df1b47717468280f64d8fe
Submitter: Jenkins
Branch: stable/havana

commit 119703b2742486e874df1b47717468280f64d8fe
Author: Michael Davies <email address hidden>
Date: Thu Oct 17 04:41:09 2013 -0700

    Fail quickly if file injection for boot volume

    Previously if the user attempted file injection on a booted volume it
    would fail and retry several times, which slowed down the instance
    spawning. Instead of attempting file injection, fail quickly, and
    log a warning

    Fix bug 1188543

    Change-Id: I264540de736c2bcb92567826fe5ba672e1244ba2
    (cherry picked from commit f59c2d64ac8f0523c587e23b2d1c59fdfe7ea131)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/54460
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a28aa07789e3b71c702657db12825b23716a4bf6
Submitter: Jenkins
Branch: stable/grizzly

commit a28aa07789e3b71c702657db12825b23716a4bf6
Author: Michael Davies <email address hidden>
Date: Thu Oct 17 04:41:09 2013 -0700

    Fail quickly if file injection for boot volume

    Previously if the user attempted file injection on a booted volume it
    would fail and retry several times, which slowed down the instance
    spawning. Instead of attempting file injection, fail quickly, and
    log a warning

    Backport to stable/grizzy

    Fix bug 1188543

    Conflicts:
     nova/virt/libvirt/driver.py

    Change-Id: I264540de736c2bcb92567826fe5ba672e1244ba2
    (cherry picked from commit f59c2d64ac8f0523c587e23b2d1c59fdfe7ea131)