nova cannot create ephemeral instance on rbd

Bug #1246708 reported by Zoltan Arnold Nagy
24
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Triaged
Low
Unassigned

Bug Description

nova.conf:

libvirt_images_type=rbd
libvirt_images_rbd_pool=ephemeraldisks

getting this:

2013-10-31 12:54:30.388 19701 TRACE nova.compute.manager [instance: 22828ed2-3bd1-4bb9-9881-4117ae503cb3] Traceback (most recent call last):
2013-10-31 12:54:30.388 19701 TRACE nova.compute.manager [instance: 22828ed2-3bd1-4bb9-9881-4117ae503cb3] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1407, in _spawn
2013-10-31 12:54:30.388 19701 TRACE nova.compute.manager [instance: 22828ed2-3bd1-4bb9-9881-4117ae503cb3] block_device_info)
2013-10-31 12:54:30.388 19701 TRACE nova.compute.manager [instance: 22828ed2-3bd1-4bb9-9881-4117ae503cb3] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2063, in spawn
2013-10-31 12:54:30.388 19701 TRACE nova.compute.manager [instance: 22828ed2-3bd1-4bb9-9881-4117ae503cb3] admin_pass=admin_password)
2013-10-31 12:54:30.388 19701 TRACE nova.compute.manager [instance: 22828ed2-3bd1-4bb9-9881-4117ae503cb3] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2450, in _create_image
2013-10-31 12:54:30.388 19701 TRACE nova.compute.manager [instance: 22828ed2-3bd1-4bb9-9881-4117ae503cb3] injection_path = image('disk').path
2013-10-31 12:54:30.388 19701 TRACE nova.compute.manager [instance: 22828ed2-3bd1-4bb9-9881-4117ae503cb3] AttributeError: 'Rbd' object has no attribute 'path'
2013-10-31 12:54:30.388 19701 TRACE nova.compute.manager [instance: 22828ed2-3bd1-4bb9-9881-4117ae503cb3]

Revision history for this message
Matt Riedemann (mriedem) wrote :

What level of openstack are you using when you hit this? Havana? Trunk?

tags: added: libvirt volumes
Revision history for this message
Jean-Sebastien Mouret (js-mouret) wrote :

Got this on Havana.

Josh Durgin (jdurgin)
tags: added: ceph rbd
Revision history for this message
Salvo Davide Rapisarda (sdrapisarda) wrote :

Havana

Revision history for this message
Salvo Davide Rapisarda (sdrapisarda) wrote :

I think it's a mysql driver bug. With sqlite driver works.

Revision history for this message
Paweł Socha (pawel-socha) wrote :

Add
libvirt_inject_partition = -2
in nova configuration on compute node.

Solly Ross (sross-7)
Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
status: Confirmed → New
importance: Medium → Undecided
Solly Ross (sross-7)
Changed in nova:
status: New → Triaged
importance: Undecided → Low
Revision history for this message
kaka (mingb1989) wrote :

I also have a similar problem in IceHouse,as follow :
<179>Sep 2 09:02:27 node-5 nova-nova.compute.manager ERROR: Instance failed to spawn
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1720, in _spawn
    block_device_info)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2467, in spawn
    admin_pass=admin_password)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2857, in _create_image
    ephemeral_size=ephemeral_gb)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 194, in cache
    *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 601, in create_image
    prepare_template(target=base, max_size=size, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 184, in fetch_func_sync
    fetch_func(target=target, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2667, in _create_ephemeral
    disk.mkfs(os_type, fs_label, target, run_as_root=is_block_dev)
  File "/usr/lib/python2.6/site-packages/nova/virt/disk/api.py", line 117, in mkfs
    utils.mkfs(default_fs, target, fs_label, run_as_root=run_as_root)
  File "/usr/lib/python2.6/site-packages/nova/utils.py", line 855, in mkfs
    execute(*args, run_as_root=run_as_root)
  File "/usr/lib/python2.6/site-packages/nova/utils.py", line 164, in execute
    return processutils.execute(*cmd, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 193, in execute
    cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mkfs -t ext3 -F -L ephemeral0 /var/lib/nova/instances/_base/ephemeral_5_default
Exit code: 1
Stdout: ''
Stderr: 'mke2fs 1.41.12 (17-May-2010)\nmkfs.ext3: No such file or directory while trying to determine filesystem size\n'

In code:

    def _create_ephemeral(self, target, ephemeral_size,
                          fs_label, os_type, is_block_dev=False,
                          max_size=None):
        if not is_block_dev:
            self._create_local(target, ephemeral_size)

        # Run as root only for block devices.
        disk.mkfs(os_type, fs_label, target, run_as_root=is_block_dev)

when use rbd "is_block_dev" is "True".so,can't create a local ephemeral disk

Revision history for this message
Yaguang Tang (heut2008) wrote :

This bug has been fixed in Juno and backported to stable/Icehouse

commit 1613cd99345e51e314bf011f47654ed730138f64
Author: Roman Podoliaka <email address hidden>
Date: Fri Aug 22 20:01:33 2014 +0300

    Fix instance boot when Ceph is used for ephemeral storage

    is_block_dev attribute was mistakenly set to True for RBD images
    (we don't actually map RBD images to block devices in the *host*
    system, but only in *guests*, so as far as the host system is
    concerned an RBD image must not be treated like a system block
    device). This led to a situation when mkfs was called for a non
    existing file and failed when trying to create an ephemeral fs.

    Closes-Bug: #1362221

    Change-Id: I54c0d117a53bb61f278b2e137bd29a595548f5a1
    (cherry picked from commit e9eb1d69bf75c22b0ad4b50e4892a5644c78cf68)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.