Out of build archive space on android-build

[Paul Sokolovsky]

$ df
/dev/sdg 516061624 515628860 432764 100% /mnt2/jenkins/jobs

That's after some clean up there already. Lacks of disk spaces causes loss of started builds, inability to start new builds.

[Paul Sokolovsky (pfalcon)]

How much time since previous disk space upgrade there? 4 months: lp:994459

[Paul Sokolovsky (pfalcon)]

I freed 5.5Gb for now, should be ok for few days, need to write proper garbage collection script.

[Paul Sokolovsky (pfalcon)]

Ok, this happens because Jenkins applies build expiration rules only when a build is made. I.e. if someone left their personal builds unattended, they will stay in Jenkins forever, even though we have a policy of personal build results to be exired in 1 month, and job has corresponding settings. This is long-standing issue, and time come to fix that.

[Paul Sokolovsky (pfalcon)]

The plan was to develop a standalone Python script to parse Jenkins job/build XML configs and apply expiration settings in them. Well, I did another pass on trying to find what other people think about the behavior of Jenkins as described above. I found http://jenkins.361315.n4.nabble.com/Is-there-a-way-to-make-quot-discard-old-builds-quot-happen-right-now-td3879634.html , which suggests a Groovy script for builtin Jenkins interpreter.

It indeed did the tricks, so the good news is that we know have 150Gb free on android-build, with much less effort than developing a standalone script. The bad news is that this is interactive feature, so has to be run manually. And there's big temptation to leave it as is, because lot of disk space was already recovered. I'm not going to give so easy though, and will investigate how hard (and insecure) will be to cron'ize that Groovy script.

[Paul Sokolovsky (pfalcon)]

The script was committed as http://bazaar.launchpad.net/~linaro-infrastructure/linaro-android-build-tools/trunk/view/head:/utils/diskspace/expire-builds.groovy , and now set up to run as a cronjob on android-build (daily).