"IOError: [Errno 28] No space left on device" on srv35-bud.bud.mirantis.net

[Dennis Dmitriev]

Please check space on srv35-bud.bud.mirantis.net, there were several failed swarm jobs:

http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.thread_1/13/
http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.centos.deploy_neutron_gre_ha.upgrade/13/
http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.centos.thread_3/13/

Logged from file iso8601.py, line 140
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 875, in emit
    self.flush()
  File "/usr/lib/python2.7/logging/__init__.py", line 835, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file iso8601.py, line 140
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 875, in emit
    self.flush()
  File "/usr/lib/python2.7/logging/__init__.py", line 835, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file iso8601.py, line 140
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 875, in emit
    self.flush()
  File "/usr/lib/python2.7/logging/__init__.py", line 835, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file iso8601.py, line 140
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 875, in emit
    self.flush()
  File "/usr/lib/python2.7/logging/__init__.py", line 835, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device

[Dennis Dmitriev (ddmitriev)]

The same issue on the srv35-bud.bud.mirantis.net , here is the list of environments:
http://paste.openstack.org/show/360903/

[Pawel Brzozowski (pbrzozowski)]

All the jobs from todays issues are currently retriggered and servers disk space have been cleaned. We are currently discussing more aggressive cleaning or rescheduling jobs to avoid that kind of issues in future.

[Pawel Brzozowski (pbrzozowski)]

Trello card created for refactoring the cleaner or using different plugin to check space before running Job on slave.

[Alexander Kurenyshev (akurenyshev)]

Today we have the same situation again. We are looking forward to get the stable fix of this issue.

[Egor Kotko (ykotko)]

Reproduced again:

http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.thread_1/42/testReport/%28root%29/prepare_slaves_3/prepare_slaves_3/

ykotko@srv35-bud:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-root 864G 814G 6.2G 100% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 32G 4.0K 32G 1% /dev
tmpfs 6.3G 15M 6.3G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 32G 0 32G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/md0 961M 67M 845M 8% /boot