add configurable workspace cleanup to Jenkins scripts

[Tim Kuhlman]

I recently discovered that our CI server was running out of disk space. On investigation in "/srv/mojo/$MOJO_PROJECT/$MOJO_SERIES" there is a ROOTFS directory and a bunch of symbolic links which point deeper in the ROOTFS, ie the symbolic link may be named "9-devel" and point to "/srv/mojo/$MOJO_PROJECT/$MOJO_SERIES/ROOTFS/srv/mojo/$MOJO_PROJECT/$MOJO_SERIES/9-devel". Looking in the ROOTFS directory it appears to be a standard LXC root image.

The primary problem is there are so many lxc images on disk, this may be related to how the Jenkins job scripts are performing or it may relate to the secondary issue of the odd lxc image in an lxc image path problem.

[Paul Collins (pjdc)]

As far as I can tell this is pretty much how LXCs and workspaces are set up in Mojo. The only difference from my local LXC-enabled projects is ROOTFS being a directory (on my machine it's a symlink back to /var/lib/lxc/mojo-foo.xenial/rootfs) but this may be something to do with how we create containers in CI and our desire to not store them in /var/lib/lxc. I'm not seeing from this description where the second LXC image comes into the picture.

I think our space problems are simply that we have too many Mojo workspaces and we never delete them. For example software-center-agent has 31 workspaces weighing in at 1.1G each even though we've only retained the last 6 runs in Jenkins.

[Paul Collins (pjdc)]

I found a mojo workspaces cleanup script deployed in the previous CI environment and it has recovered 150G (and counting) on the current environment. I don't think there's a Mojo bug here, at least w/r/t containers.

[Tim Kuhlman (timkuhlman)]

That is good that there doesn't seem to be a bug regarding containers, I do think we should still consider this a bug for the Jenkins scripts. The need to manually run a special cleanup script is a bug in my opinion. The Jenkins scripts should include a workspace cleanup step whether that is cleanup done every run or only when workspace numbers exceed a threshold either way it should happen in normal running without manual SRE intervention.