toci_quickstart.sh: recording the size of logs directory can fail with "sort: write error"

Bug #1731289 reported by Attila Darazs on 2017-11-09
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Medium
Unassigned

Bug Description

Example:

http://logs.openstack.org/48/518548/1/gate/legacy-tripleo-ci-centos-7-undercloud-oooq/da2d936/job-output.txt.gz#_2017-11-09_00_40_50_055344

2017-11-09 00:40:49.970367 | primary | +(./toci_quickstart.sh:109): du -L -ch /home/zuul/workspace/logs/ara_oooq /home/zuul/workspace/logs/delorean_logs /home/zuul/workspace/logs/devstack-gate-setup-host.txt /home/zuul/workspace/logs/devstack-gate-setup-workspace-new.txt /home/zuul/workspace/logs/full_logs.html /home/zuul/workspace/logs/postci.txt.gz /home/zuul/workspace/logs/quickstart_collect_logs.log /home/zuul/workspace/logs/quickstart_install.log /home/zuul/workspace/logs/README-logs.html /home/zuul/workspace/logs/reproduce.sh /home/zuul/workspace/logs/testrepository.subunit.gz /home/zuul/workspace/logs/undercloud
2017-11-09 00:40:49.970509 | primary | +(./toci_quickstart.sh:109): sort -rh
2017-11-09 00:40:49.970575 | primary | +(./toci_quickstart.sh:109): head -n 200
2017-11-09 00:40:50.055214 | primary | sort: write failed: standard output: Broken pipe
2017-11-09 00:40:50.055344 | primary | sort: write error

This is the problematic line: https://github.com/openstack-infra/tripleo-ci/blob/701ff27cfb11897a0c998ea6d3556fa94bfd5634/toci_quickstart.sh#L109

My suspicion is that the sorting happens in the /tmp directory which could get full (or has some other problem?) and the sort operation fails.

Possible solutions are:

* using `sort -T <dir>` and point it somewhere where there is more space.
* reduce the amount of files that we check by using `du -d 1` for example

Tags: ci Edit Tag help
Attila Darazs (adarazs) on 2017-11-09
Changed in tripleo:
importance: High → Medium
description: updated
wes hayutin (weshayutin) wrote :

This error does not impact the success of the job FYI

wes hayutin (weshayutin) on 2017-11-09
Changed in tripleo:
milestone: none → queens-3
wes hayutin (weshayutin) wrote :

need to look at this again, but this was an attempt to resolve the error
https://review.openstack.org/#/c/491884/

Changed in tripleo:
milestone: queens-3 → queens-rc1
Changed in tripleo:
milestone: queens-rc1 → rocky-1
Changed in tripleo:
milestone: rocky-1 → rocky-2
Changed in tripleo:
milestone: rocky-2 → rocky-3
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers