Failed to create lxc instance with the warning Failed to get host power management capabilities

[damitha kumarage]

I have installed openstack Essex(Installed from apt repositories this week) in a 8GM memory Ubuntu 12.04 machine. I am creating lxc instances in this setup. My instances are of type tiny(512mb memory). So I can create 14 instances in my setup.

When I try to delete my instances I get following error. When I execute nova list command I can see that that the ip's assigned to the deleted instance is detached, but the entry is still shown in the instance list.

2012-10-13 08:22:52 ERROR nova.rpc.amqp [req-f428af90-13bb-4ad8-9e91-971eee505ce4 30f21b95eb9043c0858f395a5486ce5f 155fac0cbc1f4761968dad90fa5a5e2f] Exception during message handling
2012-10-13 08:22:52 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 253, in _process_data
2012-10-13 08:22:52 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-10-13 08:22:52 TRACE nova.rpc.amqp return f(*args, **kw)
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 159, in decorated_function
2012-10-13 08:22:52 TRACE nova.rpc.amqp function(self, context, instance_uuid, *args, **kwargs)
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 183, in decorated_function
2012-10-13 08:22:52 TRACE nova.rpc.amqp sys.exc_info())
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-10-13 08:22:52 TRACE nova.rpc.amqp self.gen.next()
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in decorated_function
2012-10-13 08:22:52 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs)
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 772, in terminate_instance
2012-10-13 08:22:52 TRACE nova.rpc.amqp do_terminate_instance()
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 945, in inner
2012-10-13 08:22:52 TRACE nova.rpc.amqp retval = f(*args, **kwargs)
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 765, in do_terminate_instance
2012-10-13 08:22:52 TRACE nova.rpc.amqp self._delete_instance(context, instance)
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 743, in _delete_instance
2012-10-13 08:22:52 TRACE nova.rpc.amqp self._shutdown_instance(context, instance, 'Terminating')
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 712, in _shutdown_instance
2012-10-13 08:22:52 TRACE nova.rpc.amqp block_device_info)
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 486, in destroy
2012-10-13 08:22:52 TRACE nova.rpc.amqp cleanup=True)
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 480, in _destroy
2012-10-13 08:22:52 TRACE nova.rpc.amqp self._cleanup(instance)
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 498, in _cleanup
2012-10-13 08:22:52 TRACE nova.rpc.amqp shutil.rmtree(target)
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/shutil.py", line 245, in rmtree
2012-10-13 08:22:52 TRACE nova.rpc.amqp rmtree(fullname, ignore_errors, onerror)
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/shutil.py", line 254, in rmtree
2012-10-13 08:22:52 TRACE nova.rpc.amqp onerror(os.rmdir, path, sys.exc_info())
2012-10-13 08:22:52 TRACE nova.rpc.amqp File "/usr/lib/python2.7/shutil.py", line 252, in rmtree
2012-10-13 08:22:52 TRACE nova.rpc.amqp os.rmdir(path)
2012-10-13 08:22:52 TRACE nova.rpc.amqp OSError: [Errno 16] Device or resource busy: '/var/lib/nova/instances/instance-00000003/rootfs'
2012-10-13 08:22:52 TRACE nova.rpc.amqp

After some experiments, I commented out the following entries in /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py

if os.path.exists(target):
    shutil.rmtree(target)

After that instances are deleted correctly(but folder entries are not removed from /usr/lib/nova/instances folder as should be expected). Now I can create and delete instances correctly.

But now I get the error messages which I have pasted below, in the following two scenarios

1) I restrict the number of instances for the project to 10. Then when I try to create 11th instance I correctly get the message that the quota is exceeded. After that I delete some instances to free the resources.
Again when I try to create an new instance I get the errors below.

2) I have allowed the number of project instances to be more than 14. Then when I try to create 15th instance I get the error messages below

I have been working with Openstack setup since first release of Essex. I can see that some of the issues(some of which our team reported as well) with lxc is fixed now in the apt repositories. I have been playing with this setup for the last few days and get this error consistantly. I have attached my nova.conf as well.

2012-10-13 08:38:07 ERROR nova.compute.manager [req-afc57402-5961-4f9e-81b5-e3475a315b3b 30f21b95eb9043c0858f395a5486ce5f 155fac0cbc1f4761968dad90fa5a5e2f] [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] Instance failed to spawn

2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] Traceback (most recent call last):
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 617, in _spawn
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] self._legacy_nw_info(network_info), block_device_info)
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] return f(*args, **kw)
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 952, in spawn
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] self._create_new_domain(xml)
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 1618, in _create_new_domain
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] domain.createWithFlags(launch_flags)
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 581, in createWithFlags
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] libvirtError: internal error guest failed to start: 2012-10-13 03:08:07.294+0000: 12917: info : libvirt version: 0.9.8
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] 2012-10-13 03:08:07.294+0000: 12917: warning : lxcCapsInit:77 : Failed to get host power management capabilities
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94]
2012-10-13 08:38:07 TRACE nova.compute.manager [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94]
2012-10-13 08:38:07 DEBUG nova.compute.manager [req-afc57402-5961-4f9e-81b5-e3475a315b3b 30f21b95eb9043c0858f395a5486ce5f 155fac0cbc1f4761968dad90fa5a5e2f] [instance: 5a749aca-83c2-47b0-afb2-71e1f4108d94] Deallocating network for instance from (pid=4003) _deallocate_network /usr/lib/python2.7/dist-packages/nova/compute/manager.py:641
2012-10-13 08:38:07 DEBUG nova.rpc.amqp [req-afc57402-5961-4f9e-81b5-e3475a315b3b 30f21b95eb9043c0858f395a5486ce5f 155fac0cbc1f4761968dad90fa5a5e2f] Making asynchronous call on network ... from (pid=4003) multicall /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:326
2012-10-13 08:38:07 DEBUG nova.rpc.amqp [req-afc57402-5961-4f9e-81b5-e3475a315b3b 30f21b95eb9043c0858f395a5486ce5f 155fac0cbc1f4761968dad90fa5a5e2f] MSG_ID is d1c770aacff342a58fa14c88ec547727 from (pid=4003) multicall /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:329
2012-10-13 08:38:09 ERROR nova.rpc.amqp [req-afc57402-5961-4f9e-81b5-e3475a315b3b 30f21b95eb9043c0858f395a5486ce5f 155fac0cbc1f4761968dad90fa5a5e2f] Exception during message handling

2012-10-13 08:38:09 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 253, in _process_data
2012-10-13 08:38:09 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-10-13 08:38:09 TRACE nova.rpc.amqp return f(*args, **kw)
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 183, in decorated_function
2012-10-13 08:38:09 TRACE nova.rpc.amqp sys.exc_info())
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-10-13 08:38:09 TRACE nova.rpc.amqp self.gen.next()
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in decorated_function
2012-10-13 08:38:09 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs)
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 676, in run_instance
2012-10-13 08:38:09 TRACE nova.rpc.amqp do_run_instance()
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 945, in inner
2012-10-13 08:38:09 TRACE nova.rpc.amqp retval = f(*args, **kwargs)
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 675, in do_run_instance
2012-10-13 08:38:09 TRACE nova.rpc.amqp self._run_instance(context, instance_uuid, **kwargs)
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 476, in _run_instance
2012-10-13 08:38:09 TRACE nova.rpc.amqp self._set_instance_error_state(context, instance_uuid)
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-10-13 08:38:09 TRACE nova.rpc.amqp self.gen.next()
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 457, in _run_instance
2012-10-13 08:38:09 TRACE nova.rpc.amqp self._deallocate_network(context, instance)
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-10-13 08:38:09 TRACE nova.rpc.amqp self.gen.next()
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 454, in _run_instance
2012-10-13 08:38:09 TRACE nova.rpc.amqp injected_files, admin_password)
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 617, in _spawn
2012-10-13 08:38:09 TRACE nova.rpc.amqp self._legacy_nw_info(network_info), block_device_info)
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-10-13 08:38:09 TRACE nova.rpc.amqp return f(*args, **kw)
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 952, in spawn
2012-10-13 08:38:09 TRACE nova.rpc.amqp self._create_new_domain(xml)
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 1618, in _create_new_domain
2012-10-13 08:38:09 TRACE nova.rpc.amqp domain.createWithFlags(launch_flags)
2012-10-13 08:38:09 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/libvirt.py", line 581, in createWithFlags
2012-10-13 08:38:09 TRACE nova.rpc.amqp if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2012-10-13 08:38:09 TRACE nova.rpc.amqp libvirtError: internal error guest failed to start: 2012-10-13 03:08:07.294+0000: 12917: info : libvirt version: 0.9.8
2012-10-13 08:38:09 TRACE nova.rpc.amqp 2012-10-13 03:08:07.294+0000: 12917: warning : lxcCapsInit:77 : Failed to get host power management capabilities
2012-10-13 08:38:09 TRACE nova.rpc.amqp

[Chuck Short (zulcss)]

Anything in the libvirt log files?

chuck

[damitha kumarage (damitha)]

Hi Chuck,
I forgot to get the libvirt logs last time. I will run this again and provide them asap

Thanks,
Damitha

[Thierry Carrez (ttx)]

We cannot solve the issue you reported without more information. Could you please provide the requested information ?

[Davanum Srinivas (DIMS) (dims-v)]

This bug lacks the necessary information to effectively reproduce and fix it, therefore it has been closed. Feel free to reopen the bug by providing the requested information and set the bug status back to ''New''.