Comment 0 for bug 1731289

Revision history for this message
Attila Darazs (adarazs) wrote :

Example of a successful gate job that only failed on the last few steps in toci_quickstart.sh:

http://logs.openstack.org/48/518548/1/gate/legacy-tripleo-ci-centos-7-undercloud-oooq/da2d936/job-output.txt.gz#_2017-11-09_00_40_50_055344

2017-11-09 00:40:49.970367 | primary | +(./toci_quickstart.sh:109): du -L -ch /home/zuul/workspace/logs/ara_oooq /home/zuul/workspace/logs/delorean_logs /home/zuul/workspace/logs/devstack-gate-setup-host.txt /home/zuul/workspace/logs/devstack-gate-setup-workspace-new.txt /home/zuul/workspace/logs/full_logs.html /home/zuul/workspace/logs/postci.txt.gz /home/zuul/workspace/logs/quickstart_collect_logs.log /home/zuul/workspace/logs/quickstart_install.log /home/zuul/workspace/logs/README-logs.html /home/zuul/workspace/logs/reproduce.sh /home/zuul/workspace/logs/testrepository.subunit.gz /home/zuul/workspace/logs/undercloud
2017-11-09 00:40:49.970509 | primary | +(./toci_quickstart.sh:109): sort -rh
2017-11-09 00:40:49.970575 | primary | +(./toci_quickstart.sh:109): head -n 200
2017-11-09 00:40:50.055214 | primary | sort: write failed: standard output: Broken pipe
2017-11-09 00:40:50.055344 | primary | sort: write error

This is the problematic line: https://github.com/openstack-infra/tripleo-ci/blob/701ff27cfb11897a0c998ea6d3556fa94bfd5634/toci_quickstart.sh#L109

My suspicion is that the sorting happens in the /tmp directory which could get full (or has some other problem?) and the sort operation fails.

Possible solutions are:

* using `sort -T <dir>` and point it somewhere where there is more space.
* reduce the amount of files that we check by using `du -d 1` for example