plug_vifs failed when compute node restarts from a crash

[Peng Gu]

My compute node crashed with a running instance, nova-compute failed to restart with the following logs:

2014-07-12 16:44:48.130 AUDIT nova.service [-] Starting compute node (version 2014.2)
2014-07-12 16:44:48.236 DEBUG nova.compute.manager [req-40b7e28f-2228-4129-9291-3b6e8dd7b14d None None] [instance: a2f70ea3-f601-448a-b
cae-7af21b753c6b] Checking state from (pid=5297) _get_power_state /opt/stack/nova/nova/compute/manager.py:1068
2014-07-12 16:44:48.252 DEBUG nova.compute.manager [req-40b7e28f-2228-4129-9291-3b6e8dd7b14d None None] [instance: a2f70ea3-f601-448a-b
cae-7af21b753c6b] Checking state from (pid=5297) _get_power_state /opt/stack/nova/nova/compute/manager.py:1068
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 346, in fire_timers
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/nova/nova/openstack/common/service.py", line 490, in run_service
    service.start()
  File "/opt/stack/nova/nova/service.py", line 164, in start
    self.manager.init_host()
  File "/opt/stack/nova/nova/compute/manager.py", line 1051, in init_host
    self._init_instance(context, instance)
  File "/opt/stack/nova/nova/compute/manager.py", line 910, in _init_instance
    self.driver.plug_vifs(instance, net_info)
  File "/opt/stack/nova-docker/novadocker/virt/docker/driver.py", line 125, in plug_vifs
    raise RuntimeError(msg.format(container_id))
RuntimeError: Cannot find any PID under container "b9b465f3dc1ff1c95ef8c7a3ed4f4d0fd17d3e8b33cfbae3299e1e34839564fe"
2014-07-12 16:44:58.843 ERROR nova.openstack.common.threadgroup [-] Cannot find any PID under container "b9b465f3dc1ff1c95ef8c7a3ed4f4d
0fd17d3e8b33cfbae3299e1e34839564fe"
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 125, in wait
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup x.wait()
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 47, in wait
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wa
it
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switc
h
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in ma
in
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/service.py", line 490, in run_service
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup service.start()
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/service.py", line 164, in start
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup self.manager.init_host()
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 1051, in init_host
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup self._init_instance(context, instance)
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 910, in _init_instance
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup self.driver.plug_vifs(instance, net_info)
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova-docker/novadocker/virt/docker/driver.py", line 125, in plug_vifs

2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup raise RuntimeError(msg.format(container_id))
2014-07-12 16:44:58.843 TRACE nova.openstack.common.threadgroup RuntimeError: Cannot find any PID under container "b9b465f3dc1ff1c95ef8c7a3ed4f4d0fd17d3
e8b33cfbae3299e1e34839564fe"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/106684

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/106684
Committed: https://git.openstack.org/cgit/stackforge/nova-docker/commit/?id=000f5f2a02e4b32f6bcbfab6a72cc67ab4ab821d
Submitter: Jenkins
Branch: master

commit 000f5f2a02e4b32f6bcbfab6a72cc67ab4ab821d
Author: gupeng <[e-mail]>
Date: Mon Jul 14 12:10:26 2014 +0800

    Split plug_vifs into plug_vifs and attach_vifs

    Driver.plug_vifs is sometimes called when the container is not
    running, such as called by _init_instance in ComputeManager.

    In this patch, plug_vifs is only responsible for creating bridges
    and preparing the peer devices, which does not require the container
    running, attach_vifs is responsible for preparing the netns and
    attaching the peer device to container.

    When starting a container, attach_vifs is called after plug_vifs.

    Change-Id: Icc5e2cebde22c10a318379d41bb4c50085b07162
    Closes-bug: #1341423