Activity log for bug #1278104

Date Who What changed Old value New value Message
2014-02-09 15:17:39 Daniel Kuffner bug added bug
2014-02-09 15:17:53 Daniel Kuffner nova: assignee Daniel Kuffner (daniel-kuffner)
2014-02-09 15:26:39 OpenStack Infra nova: status New In Progress
2014-02-09 16:27:49 Daniel Kuffner description Cannot create new instances because of an internal error. It seems like that the docker client returns None instead of a empty array if not a single container is started. 2014-02-09 15:01:44.077 ERROR nova.openstack.common.threadgroup [-] 'NoneType' object is not iterable 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup Traceback (most recent call last): 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 117, in wait 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup x.wait() 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 49, in wait 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup return self.thread.wait() 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup return self._exit_event.wait() 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch() 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup return self.greenlet.switch() 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs) 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/service.py", line 480, in run_service 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup service.start() 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/service.py", line 177, in start 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup self.manager.init_host() 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 821, in init_host 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup self._destroy_evacuated_instances(context) 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 532, in _destroy_evacuated_instances 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup local_instances = self._get_instances_on_driver(context, filters) 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 511, in _get_instances_on_driver 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup driver_instances = self.driver.list_instances() 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/virt/docker/driver.py", line 96, in list_instances 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup for container in self.docker.list_containers(): 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup TypeError: 'NoneType' object is not iterable 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup Cannot create new instances because of an internal error. It seems like that the docker client returns None instead of a empty array if not a single container is started. Update: The root cause for the issue is that the docker v1.8 rest API doesn't longer deliver the container list for v1.4 api calls. We must upgrade to >= 1.7 in order to make the docker driver working again. Additional does docker not set the Content-Type correctly. The docker client expects that the Content-Type is application/json but it is plain/text. The parsing of the content will be skipped in this case Stacktrace: 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 821, in init_host 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup self._destroy_evacuated_instances(context) 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 532, in _destroy_evacuated_instances 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup local_instances = self._get_instances_on_driver(context, filters) 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 511, in _get_instances_on_driver 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup driver_instances = self.driver.list_instances() 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/virt/docker/driver.py", line 96, in list_instances 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup for container in self.docker.list_containers(): 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup TypeError: 'NoneType' object is not iterable 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup
2014-02-09 16:31:58 Daniel Kuffner description Cannot create new instances because of an internal error. It seems like that the docker client returns None instead of a empty array if not a single container is started. Update: The root cause for the issue is that the docker v1.8 rest API doesn't longer deliver the container list for v1.4 api calls. We must upgrade to >= 1.7 in order to make the docker driver working again. Additional does docker not set the Content-Type correctly. The docker client expects that the Content-Type is application/json but it is plain/text. The parsing of the content will be skipped in this case Stacktrace: 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 821, in init_host 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup self._destroy_evacuated_instances(context) 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 532, in _destroy_evacuated_instances 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup local_instances = self._get_instances_on_driver(context, filters) 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 511, in _get_instances_on_driver 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup driver_instances = self.driver.list_instances() 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/virt/docker/driver.py", line 96, in list_instances 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup for container in self.docker.list_containers(): 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup TypeError: 'NoneType' object is not iterable 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup Cannot create new instances because of an internal error. It seems like that the docker client returns None instead of a empty array if not a single container is started. Update: The root cause for the issue is that the docker v1.8 rest API doesn't longer deliver the container list for v1.4 api calls. We must upgrade to >= 1.7 in order to make the docker driver working again. Additional does docker not set the Content-Type correctly. The docker client expects that the Content-Type is application/json but it is plain/text. The parsing of the content will be skipped in this case. Please see also: https://github.com/dotcloud/docker/pull/3974 Stacktrace: 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 821, in init_host 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup self._destroy_evacuated_instances(context) 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 532, in _destroy_evacuated_instances 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup local_instances = self._get_instances_on_driver(context, filters) 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 511, in _get_instances_on_driver 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup driver_instances = self.driver.list_instances() 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/virt/docker/driver.py", line 96, in list_instances 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup for container in self.docker.list_containers(): 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup TypeError: 'NoneType' object is not iterable 2014-02-09 15:01:44.077 TRACE nova.openstack.common.threadgroup
2014-02-11 03:03:40 OpenStack Infra nova: status In Progress Fix Committed
2014-02-21 14:46:55 Russell Bryant nova: milestone icehouse-3
2014-03-05 13:12:37 Thierry Carrez nova: status Fix Committed Fix Released
2014-04-17 09:08:04 Thierry Carrez nova: milestone icehouse-3 2014.1