logstash query: message:"cgroup\: No space left on device" AND filename:logs*screen-n-cpu.txt
http://logs.openstack.org/12/80412/8/check/check-tempest-dsvm-postgres-full/f9f6158/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-03-21_17_45_12_490
ERROR nova.compute.manager [req-630b71d6-0fbe-4e9e-99fe-019da7d29a3a FixedIPsNegativeTestJson-475659359 FixedIPsNegativeTestJson-265680949] [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] Error: internal error unable to add domain instance-00000002 task 3057 to cgroup: No space left on device
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] Traceback (most recent call last):
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/compute/manager.py", line 1304, in _build_instance
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] set_access_ip=set_access_ip)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/compute/manager.py", line 394, in decorated_function
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] return function(self, context, *args, **kwargs)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/compute/manager.py", line 1716, in _spawn
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] LOG.exception(_('Instance failed to spawn'), instance=instance)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 68, in __exit__
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] six.reraise(self.type_, self.value, self.tb)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/compute/manager.py", line 1713, in _spawn
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] block_device_info)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2241, in spawn
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] block_device_info)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3621, in _create_domain_and_network
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] power_on=power_on)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3531, in _create_domain
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] domain.XMLDesc(0))
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 68, in __exit__
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] six.reraise(self.type_, self.value, self.tb)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3526, in _create_domain
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] domain.createWithFlags(launch_flags)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] result = proxy_call(self._autowrap, f, *args, **kwargs)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] rv = execute(f,*args,**kwargs)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] rv = meth(*args,**kwargs)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 581, in createWithFlags
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] libvirtError: internal error unable to add domain instance-00000002 task 3057 to cgroup: No space left on device
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] Traceback (most recent call last):
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/compute/manager.py", line 1184, in _run_instance
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] instance, image_meta, legacy_bdm_in_spec)
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/compute/manager.py", line 1354, in _build_instance
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] reason=unicode(exc_info[1]))
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] RescheduledException: Build of instance 3f281136-ed69-4bfb-bf36-a7d4aa1c0640 was re-scheduled: internal error unable to add domain instance-00000002 task 3057 to cgroup: No space left on device
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]
This started right around the time we started using: https://review.openstack.org/#/c/79816/ (serge's libvirt fix for https://bugs.launchpad.net/nova/+bug/1254872)
We have seen 10 hits in last 12 hours across all jobs, so this bug appears to be less frequent then https:/ /bugs.launchpad .net/nova/ +bug/1254872 was