Server won't cleanup its /tmp mess after importing images

[Caio Begotti]

The server's import-image util should remove all temp files it produces to generate diffs of the images imported after it has finished executing. I heard the other day the main production server of system-image.u.c got its root disk filled up and I bet this bug helped with that. It filled up my root disk also...

For the moment a cronjob is taking care of removing all these /tmp/tmp* files after import-image runs, but ideally import-image should clean up its own mess.

Well, even more ideally it should use a temp dir inside of base_path as system-image is usually in a volume disk on production servers so it would be quite safe disk space-wise. The system's /tmp should be avoided.

This is from a pretty low usage server we have in PES, just to give you an idea how of much disk space is wasted in a few days:

# du -sh tmp*
8.0K tmp0igyi_
20K tmp1lO7Of
8.0K tmp1Wn3s2
7.6M tmp53aRpC
4.0K tmp5Bp_ZV
4.0K tmp7FaBem
2.0M tmp7FMWU7
4.0K tmp7Pyw8S
8.0K tmp8CAfZI
4.0K tmp9iVaXp
4.0K tmp9KFulm
12K tmpA04dcR
4.0K tmpal5qjx
8.0K tmpAOhIbl
533M tmpbIwWlZ
4.0K tmpbuJHcD
1.1G tmpChkuKd
4.0K tmpDl5YWw
16K tmpDngHkl
4.0K tmpgh5s99
4.0K tmpGNUA72
4.0K tmpgXWVXM
4.0K tmpHjzWG4
516K tmpHPHStG
1.1G tmphyPFkg
4.0K tmpilQe5O
12K tmpIxRQ_R
4.0K tmpjdWDuh
4.0K tmpJsu_PY
8.0K tmpJY1qkW
264K tmpJymQKm
1.1G tmpkLgwUe
4.0K tmpL4NgR9
12K tmpLTuYA1
4.0K tmpLzsmkI
1020K tmpmMYF8o
4.0K tmpMWL96Z
4.0K tmpn6ixNh
20K tmpn9wAUq
20K tmpNPxJdP
4.0K tmppbS2zJ
4.0K tmppiT6vi
4.0K tmpPjBtW5
4.0K tmpQ0CvC6
1.1G tmpqJXOaZ
4.0K tmprwnAFz
4.0K tmpsqDTAe
4.0K tmpSWo5IK
120K tmpTaBsz3
8.0K tmpTLLyiJ
4.0K tmpTrBNlp
68K tmpTxvbGp
4.0K tmpufBJRu
4.0K tmpUQtYic
1.1G tmpvRjr2I
4.0K tmpWbDTO8
1.1G tmpWZ1NBA
3.9M tmpX_FknX
4.0K tmpzBvspZ
4.0K tmpzSIirt
35M tmpZTFqmK
8.0K tmpzXGMs2

[Steve Langasek (vorlon)]

On Wed, Jul 22, 2015 at 02:32:21PM -0000, Caio Begotti wrote:
> Public bug reported:

> The server's import-image util should remove all temp files it produces
> to generate diffs of the images imported after it has finished
> executing. I heard the other day the main production server of system-
> image.u.c got its root disk filled up and I bet this bug helped with
> that. It filled up my root disk also...

I think this may have been talking about something else. I don't find any
mention of nusakan's root disk filling up in #is-outage, and we run
import-images with TMPDIR=/srv/system-image.ubuntu.com/tmp set. I think
it's reasonable to have this be the responsibility of the caller rather than
hard-coding such behavior in the code.

> For the moment a cronjob is taking care of removing all these /tmp/tmp*
> files after import-image runs, but ideally import-image should clean up
> its own mess.

In fact, we don't see this behavior here. We do have some tmp files left
behind:

$ du -sh tmp/
2.8G tmp/
$

This is 15 tmp directories that all date to a two-hour window on July 17. I
don't know what was happening on the server then but clearly import-images
was failing abnormally. But as you can see, the vast majority of
import-image runs are cleaning up correctly.

Yes, we should make sure the files are always cleaned up, but we'll need
some idea of how to reproduce this. If you have logs corresponding to the
timestamps of the files left behind, that might be a good place to start.

[Caio Begotti (caio1982)]

Hey! So, first of all, I didn't know we could set TMPDIR :-) I'll surely try it. Thanks, Steve!

I don't remember exactly when the root disk of the main server got filled up but it was last week or so? Anyway, I saw that happening yesterday when IS moved our system-image server over to the new production environment, which I suppose for some reason triggered import-images to reprocess lots of files.

We don't really have access to those temps anymore as IS is now pruning them from time to time in a cronjob :-(

Given production machines are all VMs now and their root disk are pretty small (and if you need plenty of space you have to use separate volumes) I think it's reasonable to have TMPDIR pre-set and make sure system-image is contained in its base_path, but that's really my personal opinion only.