My compute node crashed with a running instance, nova-compute failed to restart whit the following logs:
2014-07-12 16:44:48.130 AUDIT nova.service [-] Starting compute node (version 2014.2) 2014-07-12 16:44:48.236 DEBUG nova.compute.manager [req-40b7e28f-2228-4129-9291-3b6e8dd7b14d None None] [instance: a2f70ea3-f601-448a-b cae-7af21b753c6b] Checking state from (pid=5297) _get_power_state /opt/stack/nova/nova/compute/manager.py:1068 2014-07-12 16:44:48.252 DEBUG nova.compute.manager [req-40b7e28f-2228-4129-9291-3b6e8dd7b14d None None] [instance: a2f70ea3-f601-448a-b cae-7af21b753c6b] Checking state from (pid=5297) _get_power_state /opt/stack/nova/nova/compute/manager.py:1068 Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 346, in fire_timers timer() File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__ cb(*args, **kw) File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main result = function(*args, **kwargs) File "/opt/stack/nova/nova/openstack/common/service.py", line 490, in run_service service.start() File "/opt/stack/nova/nova/service.py", line 164, in start self.manager.init_host() File "/opt/stack/nova/nova/compute/manager.py", line 1051, in init_host self._init_instance(context, instance) File "/opt/stack/nova/nova/compute/manager.py", line 910, in _init_instance self.driver.plug_vifs(instance, net_info) File "/opt/stack/nova-docker/novadocker/virt/docker/driver.py", line 125, in plug_vifs raise RuntimeError(msg.format(container_id)) RuntimeError: Cannot find any PID under container "b9b465f3dc1ff1c95ef8c7a3ed4f4d0fd17d3e8b33cfbae3299e1e34839564fe" 2014-07-12 16:44:58.843 ERROR nova.openstack.common.threadgroup [-] Cannot find any PID under container "b9b465f3dc1ff1c95ef8c7a3ed4f4d 0fd17d3e8b33cfbae3299e1e34839564fe" 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup Traceback (most recent call last): 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 125, in wait 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup x.wait() 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 47, in wait 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup return self.thread.wait() 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wa it 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup return self._exit_event.wait() 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch() 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switc h 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup return self.greenlet.switch() 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in ma in 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs) 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/service.py", line 490, in run_service 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup service.start() 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/service.py", line 164, in start 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup self.manager.init_host() 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 1051, in init_host 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup self._init_instance(context, instance) 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 910, in _init_instance 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup self.driver.plug_vifs(instance, net_info) 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova-docker/novadocker/virt/docker/driver.py", line 125, in plug_vifs
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup raise RuntimeError(msg.format(container_id)) 2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup RuntimeError: Cannot find any PID under container "b9b465f3dc1ff1c95ef8c7a3ed4f4d0fd17d3 e8b33cfbae3299e1e34839564fe"
My compute node crashed with a running instance, nova-compute failed to restart whit the following logs:
2014-07-12 16:44:48.130 AUDIT nova.service [-] Starting compute node (version 2014.2) manager [req-40b7e28f- 2228-4129- 9291-3b6e8dd7b1 4d None None] [instance: a2f70ea3- f601-448a- b nova/nova/ compute/ manager. py:1068 manager [req-40b7e28f- 2228-4129- 9291-3b6e8dd7b1 4d None None] [instance: a2f70ea3- f601-448a- b nova/nova/ compute/ manager. py:1068 lib/python2. 7/dist- packages/ eventlet/ hubs/hub. py", line 346, in fire_timers lib/python2. 7/dist- packages/ eventlet/ hubs/timer. py", line 56, in __call__ lib/python2. 7/dist- packages/ eventlet/ greenthread. py", line 194, in main nova/nova/ openstack/ common/ service. py", line 490, in run_service nova/nova/ service. py", line 164, in start manager. init_host( ) nova/nova/ compute/ manager. py", line 1051, in init_host _init_instance( context, instance) nova/nova/ compute/ manager. py", line 910, in _init_instance driver. plug_vifs( instance, net_info) nova-docker/ novadocker/ virt/docker/ driver. py", line 125, in plug_vifs msg.format( container_ id)) c95ef8c7a3ed4f4 d0fd17d3e8b33cf bae3299e1e34839 564fe" common. threadgroup [-] Cannot find any PID under container "b9b465f3dc1ff1 c95ef8c7a3ed4f4 d ae3299e1e348395 64fe" common. threadgroup Traceback (most recent call last): common. threadgroup File "/opt/stack/ nova/nova/ openstack/ common/ threadgroup. py", line 125, in wait common. threadgroup x.wait() common. threadgroup File "/opt/stack/ nova/nova/ openstack/ common/ threadgroup. py", line 47, in wait common. threadgroup return self.thread.wait() common. threadgroup File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ greenthread. py", line 168, in wa common. threadgroup return self._exit_ event.wait( ) common. threadgroup File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ event.py" , line 116, in wait common. threadgroup return hubs.get_ hub().switch( ) common. threadgroup File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ hubs/hub. py", line 187, in switc common. threadgroup return self.greenlet. switch( ) common. threadgroup File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ greenthread. py", line 194, in ma common. threadgroup result = function(*args, **kwargs) common. threadgroup File "/opt/stack/ nova/nova/ openstack/ common/ service. py", line 490, in run_service common. threadgroup service.start() common. threadgroup File "/opt/stack/ nova/nova/ service. py", line 164, in start common. threadgroup self.manager. init_host( ) common. threadgroup File "/opt/stack/ nova/nova/ compute/ manager. py", line 1051, in init_host common. threadgroup self._init_ instance( context, instance) common. threadgroup File "/opt/stack/ nova/nova/ compute/ manager. py", line 910, in _init_instance common. threadgroup self.driver. plug_vifs( instance, net_info) common. threadgroup File "/opt/stack/ nova-docker/ novadocker/ virt/docker/ driver. py", line 125, in plug_vifs
2014-07-12 16:44:48.236 DEBUG nova.compute.
cae-7af21b753c6b] Checking state from (pid=5297) _get_power_state /opt/stack/
2014-07-12 16:44:48.252 DEBUG nova.compute.
cae-7af21b753c6b] Checking state from (pid=5297) _get_power_state /opt/stack/
Traceback (most recent call last):
File "/usr/local/
timer()
File "/usr/local/
cb(*args, **kw)
File "/usr/local/
result = function(*args, **kwargs)
File "/opt/stack/
service.start()
File "/opt/stack/
self.
File "/opt/stack/
self.
File "/opt/stack/
self.
File "/opt/stack/
raise RuntimeError(
RuntimeError: Cannot find any PID under container "b9b465f3dc1ff1
2014-07-12 16:44:58.843 ERROR nova.openstack.
0fd17d3e8b33cfb
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
it
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
h
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
in
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack.
2014-07-12 16:44:58.843 TRACE nova.openstack. common. threadgroup raise RuntimeError( msg.format( container_ id)) common. threadgroup RuntimeError: Cannot find any PID under container "b9b465f3dc1ff1 c95ef8c7a3ed4f4 d0fd17d3 1e34839564fe"
2014-07-12 16:44:58.843 TRACE nova.openstack.
e8b33cfbae3299e