| 2016-01-20 17:08:39 |
Thomas Maddox |
bug |
|
|
added bug |
| 2016-01-20 18:53:13 |
Serge Hallyn |
bug task added |
|
linux (Ubuntu) |
|
| 2016-01-20 18:57:33 |
Thomas Maddox |
description |
There appears to be a regression from kernel 3.13.0-58-generic where virDomainShutdown no longer brings the domain to shutoff state in newer Ubuntu kernels.
Details:
OS:
$ lsb_release -rd
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Kernel:
$ uname -a
Linux thomas-devstack01 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Package:
$ dpkg -l | grep libvirt-bin
ii libvirt-bin 1.2.2-0ubuntu13.1.16 amd64 programs for the libvirt library
Kernel: 3.13.0-76-generic
Libvirt Version: 1.2.2
Libvirt virt driver: LXC
Issue seen from console:
$ sudo virsh -c lxc:/// shutdown instance-00000001
error: Failed to shutdown domain instance-00000001
error: internal error: Child process (1304) unexpected exit status 1
libvirtd logs:
2016-01-20 16:42:38.267+0000: 14089: debug : virDomainShutdown:3094 : dom=0x7f89d0004a70, (VM: name=instance-00000001, uuid=e342beb8-8666-4a47-8b40-5844e866e1f7)
2016-01-20 16:42:38.269+0000: 14089: error : virProcessWait:191 : internal error: Child process (1304) unexpected exit status 1
nova-compute logs (when attempting to do a nova stop <instance> in devstack):
2016-01-20 16:37:35.244 ERROR oslo_messaging.rpc.dispatcher [req-dd0fe69c-9bd2-4de5-b59b-77619699d69b demo demo] Exception during message handling: internal error: Child process
(631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher payload)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 357, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 407, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 385, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 373, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2533, in stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher do_stop_instance()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2525, in do_stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._power_off_instance(context, instance, clean_shutdown)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2260, in _power_off_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self.driver.power_off(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2376, in power_off
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._clean_shutdown(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2331, in _clean_shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher guest.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 451, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._domain.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = execute(f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(c, e, tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = meth(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 2172, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher libvirtError: internal error: Child process (631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher
This all worked gracefully on 3.13.0-58-generic, as well as the newer 3.13.x kernels, including upstream 3.13.0 installed from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-trusty/linux-image-3.13.0-031300-generic_3.13.0-031300.201401192235_amd64.deb. |
There appears to be a regression from kernel 3.13.0-58-generic where virDomainShutdown no longer brings the domain to shutoff state in newer Ubuntu kernels.
Details:
========
OS:
---
$ lsb_release -rd
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Kernel:
-------
$ uname -a
Linux thomas-devstack01 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Package:
--------
$ dpkg -l | grep libvirt-bin
ii libvirt-bin 1.2.2-0ubuntu13.1.16 amd64 programs for the libvirt library
Libvirt virt driver: LXC
Issue seen from console:
------------------------
$ sudo virsh -c lxc:/// shutdown instance-00000001
error: Failed to shutdown domain instance-00000001
error: internal error: Child process (1304) unexpected exit status 1
libvirtd logs:
--------------
2016-01-20 16:42:38.267+0000: 14089: debug : virDomainShutdown:3094 : dom=0x7f89d0004a70, (VM: name=instance-00000001, uuid=e342beb8-8666-4a47-8b40-5844e866e1f7)
2016-01-20 16:42:38.269+0000: 14089: error : virProcessWait:191 : internal error: Child process (1304) unexpected exit status 1
nova-compute logs
-----------------
*Note*: From when attempting to do a nova stop <instance> in devstack
2016-01-20 16:37:35.244 ERROR oslo_messaging.rpc.dispatcher [req-dd0fe69c-9bd2-4de5-b59b-77619699d69b demo demo] Exception during message handling: internal error: Child process
(631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher payload)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 357, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 407, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 385, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 373, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2533, in stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher do_stop_instance()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2525, in do_stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._power_off_instance(context, instance, clean_shutdown)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2260, in _power_off_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self.driver.power_off(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2376, in power_off
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._clean_shutdown(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2331, in _clean_shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher guest.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 451, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._domain.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = execute(f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(c, e, tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = meth(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 2172, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher libvirtError: internal error: Child process (631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher
This all worked gracefully on 3.13.0-58-generic, as well as the newer 3.13.x kernels, including upstream 3.13.0 installed from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-trusty/linux-image-3.13.0-031300-generic_3.13.0-031300.201401192235_amd64.deb. |
|
| 2016-01-20 18:59:11 |
Thomas Maddox |
description |
There appears to be a regression from kernel 3.13.0-58-generic where virDomainShutdown no longer brings the domain to shutoff state in newer Ubuntu kernels.
Details:
========
OS:
---
$ lsb_release -rd
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Kernel:
-------
$ uname -a
Linux thomas-devstack01 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Package:
--------
$ dpkg -l | grep libvirt-bin
ii libvirt-bin 1.2.2-0ubuntu13.1.16 amd64 programs for the libvirt library
Libvirt virt driver: LXC
Issue seen from console:
------------------------
$ sudo virsh -c lxc:/// shutdown instance-00000001
error: Failed to shutdown domain instance-00000001
error: internal error: Child process (1304) unexpected exit status 1
libvirtd logs:
--------------
2016-01-20 16:42:38.267+0000: 14089: debug : virDomainShutdown:3094 : dom=0x7f89d0004a70, (VM: name=instance-00000001, uuid=e342beb8-8666-4a47-8b40-5844e866e1f7)
2016-01-20 16:42:38.269+0000: 14089: error : virProcessWait:191 : internal error: Child process (1304) unexpected exit status 1
nova-compute logs
-----------------
*Note*: From when attempting to do a nova stop <instance> in devstack
2016-01-20 16:37:35.244 ERROR oslo_messaging.rpc.dispatcher [req-dd0fe69c-9bd2-4de5-b59b-77619699d69b demo demo] Exception during message handling: internal error: Child process
(631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher payload)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 357, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 407, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 385, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 373, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2533, in stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher do_stop_instance()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2525, in do_stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._power_off_instance(context, instance, clean_shutdown)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2260, in _power_off_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self.driver.power_off(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2376, in power_off
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._clean_shutdown(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2331, in _clean_shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher guest.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 451, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._domain.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = execute(f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(c, e, tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = meth(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 2172, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher libvirtError: internal error: Child process (631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher
This all worked gracefully on 3.13.0-58-generic, as well as the newer 3.13.x kernels, including upstream 3.13.0 installed from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-trusty/linux-image-3.13.0-031300-generic_3.13.0-031300.201401192235_amd64.deb. |
There appears to be a regression from kernel 3.13.0-58-generic where virDomainShutdown no longer brings the domain to shutoff state in newer Ubuntu kernels.
Details:
========
OS:
---
$ lsb_release -rd
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Kernel:
-------
$ uname -a
Linux thomas-devstack01 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Package:
--------
$ dpkg -l | grep libvirt-bin
ii libvirt-bin 1.2.2-0ubuntu13.1.16 amd64 programs for the libvirt library
Libvirt virt driver: LXC
Issue seen from console:
------------------------
$ sudo virsh -c lxc:/// shutdown instance-00000001
error: Failed to shutdown domain instance-00000001
error: internal error: Child process (1304) unexpected exit status 1
libvirtd logs:
--------------
2016-01-20 16:42:38.267+0000: 14089: debug : virDomainShutdown:3094 : dom=0x7f89d0004a70, (VM: name=instance-00000001, uuid=e342beb8-8666-4a47-8b40-5844e866e1f7)
2016-01-20 16:42:38.269+0000: 14089: error : virProcessWait:191 : internal error: Child process (1304) unexpected exit status 1
nova-compute logs
-----------------
*Note*: From when attempting to do a nova stop <instance> in devstack
```
2016-01-20 16:37:35.244 ERROR oslo_messaging.rpc.dispatcher [req-dd0fe69c-9bd2-4de5-b59b-77619699d69b demo demo] Exception during message handling: internal error: Child process
(631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher payload)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 357, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 407, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 385, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 373, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2533, in stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher do_stop_instance()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2525, in do_stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._power_off_instance(context, instance, clean_shutdown)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2260, in _power_off_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self.driver.power_off(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2376, in power_off
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._clean_shutdown(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2331, in _clean_shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher guest.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 451, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._domain.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = execute(f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(c, e, tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = meth(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 2172, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher libvirtError: internal error: Child process (631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher
```
This all worked gracefully on 3.13.0-58-generic, as well as the newer 3.13.x kernels, including upstream 3.13.0 installed from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-trusty/linux-image-3.13.0-031300-generic_3.13.0-031300.201401192235_amd64.deb. |
|
| 2016-01-20 19:00:16 |
Brad Figg |
linux (Ubuntu): status |
New |
Incomplete |
|
| 2016-01-20 19:00:17 |
Brad Figg |
tags |
libvirt lxc |
libvirt lxc trusty |
|
| 2016-01-20 19:00:19 |
Thomas Maddox |
description |
There appears to be a regression from kernel 3.13.0-58-generic where virDomainShutdown no longer brings the domain to shutoff state in newer Ubuntu kernels.
Details:
========
OS:
---
$ lsb_release -rd
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Kernel:
-------
$ uname -a
Linux thomas-devstack01 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Package:
--------
$ dpkg -l | grep libvirt-bin
ii libvirt-bin 1.2.2-0ubuntu13.1.16 amd64 programs for the libvirt library
Libvirt virt driver: LXC
Issue seen from console:
------------------------
$ sudo virsh -c lxc:/// shutdown instance-00000001
error: Failed to shutdown domain instance-00000001
error: internal error: Child process (1304) unexpected exit status 1
libvirtd logs:
--------------
2016-01-20 16:42:38.267+0000: 14089: debug : virDomainShutdown:3094 : dom=0x7f89d0004a70, (VM: name=instance-00000001, uuid=e342beb8-8666-4a47-8b40-5844e866e1f7)
2016-01-20 16:42:38.269+0000: 14089: error : virProcessWait:191 : internal error: Child process (1304) unexpected exit status 1
nova-compute logs
-----------------
*Note*: From when attempting to do a nova stop <instance> in devstack
```
2016-01-20 16:37:35.244 ERROR oslo_messaging.rpc.dispatcher [req-dd0fe69c-9bd2-4de5-b59b-77619699d69b demo demo] Exception during message handling: internal error: Child process
(631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher payload)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 357, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 407, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 385, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 373, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2533, in stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher do_stop_instance()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2525, in do_stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._power_off_instance(context, instance, clean_shutdown)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2260, in _power_off_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self.driver.power_off(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2376, in power_off
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._clean_shutdown(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2331, in _clean_shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher guest.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 451, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._domain.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = execute(f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(c, e, tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = meth(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 2172, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher libvirtError: internal error: Child process (631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher
```
This all worked gracefully on 3.13.0-58-generic, as well as the newer 3.13.x kernels, including upstream 3.13.0 installed from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-trusty/linux-image-3.13.0-031300-generic_3.13.0-031300.201401192235_amd64.deb. |
There appears to be a regression from kernel 3.13.0-58-generic where virDomainShutdown no longer brings the domain to shutoff state in newer Ubuntu kernels.
Details:
========
OS:
---
$ lsb_release -rd
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Kernel:
-------
$ uname -a
Linux thomas-devstack01 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Package:
--------
$ dpkg -l | grep libvirt-bin
ii libvirt-bin 1.2.2-0ubuntu13.1.16 amd64 programs for the libvirt library
Libvirt virt driver: LXC
Issue seen from console:
------------------------
$ sudo virsh -c lxc:/// shutdown instance-00000001
error: Failed to shutdown domain instance-00000001
error: internal error: Child process (1304) unexpected exit status 1
libvirtd logs:
--------------
2016-01-20 16:42:38.267+0000: 14089: debug : virDomainShutdown:3094 : dom=0x7f89d0004a70, (VM: name=instance-00000001, uuid=e342beb8-8666-4a47-8b40-5844e866e1f7)
2016-01-20 16:42:38.269+0000: 14089: error : virProcessWait:191 : internal error: Child process (1304) unexpected exit status 1
nova-compute logs
-----------------
*Note*: From when attempting to do a nova stop <instance> in devstack
2016-01-20 16:37:35.244 ERROR oslo_messaging.rpc.dispatcher [req-dd0fe69c-9bd2-4de5-b59b-77619699d69b demo demo] Exception during message handling: internal error: Child process
(631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher payload)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 357, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 407, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 385, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 373, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2533, in stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher do_stop_instance()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2525, in do_stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._power_off_instance(context, instance, clean_shutdown)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2260, in _power_off_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self.driver.power_off(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2376, in power_off
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._clean_shutdown(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2331, in _clean_shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher guest.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 451, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._domain.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = execute(f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(c, e, tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = meth(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 2172, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher libvirtError: internal error: Child process (631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher
This all worked gracefully on 3.13.0-58-generic, as well as the newer 3.13.x kernels, including upstream 3.13.0 installed from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-trusty/linux-image-3.13.0-031300-generic_3.13.0-031300.201401192235_amd64.deb. |
|
| 2016-01-20 19:23:00 |
Thomas Maddox |
linux (Ubuntu): status |
Incomplete |
Confirmed |
|
| 2016-01-20 20:05:39 |
Launchpad Janitor |
libvirt (Ubuntu): status |
New |
Confirmed |
|
| 2016-01-20 20:49:32 |
Corey O'Brien |
bug |
|
|
added subscriber Corey O'Brien |
| 2016-01-26 03:52:32 |
penalvch |
linux (Ubuntu): importance |
Undecided |
Medium |
|
