Unable to pre-create chardev file when booting from volume

Bug #1123274 reported by Boris Deschenes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Nikola Đipanov

Bug Description

I am using openstack-ubuntu-testing-grizzly and I have this problem when booting from volumes, this was working well before but seems to have broken in the last 72 hours.

booting from local storage works fine but when trying to boot from volume, libvirt complains that the instance-xxxxxxxx directory does not exist:

2013-02-12 12:36:05.860 ERROR nova.compute.manager [req-808284f5-5641-4fd6-b8fd-733f243e49da 8636766bc6b342539a9c1b0e3e0c24cc 24e6c25ee7bf4f5bbcd19c53bbe84f37] [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] Instance failed to spawn
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] Traceback (most recent call last):
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 985, in _spawn
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] block_device_info)
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1200, in spawn
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] block_device_info)
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2082, in _create_domain_and_network
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] domain = self._create_domain(xml, instance=instance)
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2043, in _create_domain
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] domain.createWithFlags(launch_flags)
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] rv = execute(f,*args,**kwargs)
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] rv = meth(*args,**kwargs)
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 650, in createWithFlags
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2013-02-12 12:36:05.860 23382 TRACE nova.compute.manager [instance: 0d17dae7-66e8-491d-8727-77ba163dfb94] libvirtError: Unable to pre-create chardev file '/var/lib/nova/instances/instance-00000122/console.log': No such file or directory

Again, this is only when booting from volume.

Boris

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :
Revision history for this message
Boris Deschenes (boris-michel-deschenes) wrote :

could very well be the same thing, this is with KVM and libvirt as well

Revision history for this message
Chuck Short (zulcss) wrote :

Can you attach your libvirt log file when this happens?

Changed in nova:
status: New → Incomplete
Revision history for this message
Boris Deschenes (boris-michel-deschenes) wrote :

I'm sorry, I don't have much logs in libvirt...

I temporarily fixed this problem by adding a manual (additional) directory creation in the spawn function of nova/virt/libvrti/driver.py (around line 1200), so my guess is that the problem is the workflow does not create the directory soon enough...

    def spawn(self, context, instance, image_meta, injected_files,
              admin_password, network_info=None, block_device_info=None):
        disk_info = blockinfo.get_disk_info(CONF.libvirt_type,
                                            instance,
                                            block_device_info,
                                            image_meta)
        xml = self.to_xml(instance, network_info,
                          disk_info, image_meta,
                          block_device_info=block_device_info)
        if image_meta:
            self._create_image(context, instance, xml,
                               disk_info['mapping'],
                               network_info=network_info,
                               block_device_info=block_device_info,
                               files=injected_files,
                               admin_pass=admin_password)
        #
        # MANUAL DIRECTORY CREATION
        #
        instance_dir = libvirt_utils.get_instance_path(instance)
        if not os.path.exists(instance_dir):
            os.mkdir(instance_dir)
        #
        # END

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/23818

Changed in nova:
assignee: nobody → Nikola Đipanov (ndipanov)
status: Incomplete → In Progress
Changed in nova:
milestone: none → grizzly-rc1
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/23818
Committed: http://github.com/openstack/nova/commit/3133096f2bf2c8de04c70c3f3209d727a4c8cfb3
Submitter: Jenkins
Branch: master

commit 3133096f2bf2c8de04c70c3f3209d727a4c8cfb3
Author: Nikola Dipanov <email address hidden>
Date: Thu Mar 7 15:54:07 2013 +0100

    Libvirt driver create images even without meta

    This patch allows the libvirt driver to call _create_image and
    create instance directory and other needed images even when there is no
    image metadata specified.

    As an added bonus, this patch refactors the code of _create_image
    method a bit to make it more clear to the reader when some of the images
    will and will not be created, especially with regard to booting from
    volume.

    Fixes bug: 1124441
    Fixes bug: 1131913
    Fixes bug: 1123274

    blueprint: improve-boot-from-volume

    Change-Id: I5a028dc0585876d35be4fb86df3a423d89e054ee

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-rc1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.