Server won't cleanup its /tmp mess after importing images
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu system image |
New
|
Undecided
|
Unassigned |
Bug Description
The server's import-image util should remove all temp files it produces to generate diffs of the images imported after it has finished executing. I heard the other day the main production server of system-image.u.c got its root disk filled up and I bet this bug helped with that. It filled up my root disk also...
For the moment a cronjob is taking care of removing all these /tmp/tmp* files after import-image runs, but ideally import-image should clean up its own mess.
Well, even more ideally it should use a temp dir inside of base_path as system-image is usually in a volume disk on production servers so it would be quite safe disk space-wise. The system's /tmp should be avoided.
This is from a pretty low usage server we have in PES, just to give you an idea how of much disk space is wasted in a few days:
# du -sh tmp*
8.0K tmp0igyi_
20K tmp1lO7Of
8.0K tmp1Wn3s2
7.6M tmp53aRpC
4.0K tmp5Bp_ZV
4.0K tmp7FaBem
2.0M tmp7FMWU7
4.0K tmp7Pyw8S
8.0K tmp8CAfZI
4.0K tmp9iVaXp
4.0K tmp9KFulm
12K tmpA04dcR
4.0K tmpal5qjx
8.0K tmpAOhIbl
533M tmpbIwWlZ
4.0K tmpbuJHcD
1.1G tmpChkuKd
4.0K tmpDl5YWw
16K tmpDngHkl
4.0K tmpgh5s99
4.0K tmpGNUA72
4.0K tmpgXWVXM
4.0K tmpHjzWG4
516K tmpHPHStG
1.1G tmphyPFkg
4.0K tmpilQe5O
12K tmpIxRQ_R
4.0K tmpjdWDuh
4.0K tmpJsu_PY
8.0K tmpJY1qkW
264K tmpJymQKm
1.1G tmpkLgwUe
4.0K tmpL4NgR9
12K tmpLTuYA1
4.0K tmpLzsmkI
1020K tmpmMYF8o
4.0K tmpMWL96Z
4.0K tmpn6ixNh
20K tmpn9wAUq
20K tmpNPxJdP
4.0K tmppbS2zJ
4.0K tmppiT6vi
4.0K tmpPjBtW5
4.0K tmpQ0CvC6
1.1G tmpqJXOaZ
4.0K tmprwnAFz
4.0K tmpsqDTAe
4.0K tmpSWo5IK
120K tmpTaBsz3
8.0K tmpTLLyiJ
4.0K tmpTrBNlp
68K tmpTxvbGp
4.0K tmpufBJRu
4.0K tmpUQtYic
1.1G tmpvRjr2I
4.0K tmpWbDTO8
1.1G tmpWZ1NBA
3.9M tmpX_FknX
4.0K tmpzBvspZ
4.0K tmpzSIirt
35M tmpZTFqmK
8.0K tmpzXGMs2
On Wed, Jul 22, 2015 at 02:32:21PM -0000, Caio Begotti wrote:
> Public bug reported:
> The server's import-image util should remove all temp files it produces
> to generate diffs of the images imported after it has finished
> executing. I heard the other day the main production server of system-
> image.u.c got its root disk filled up and I bet this bug helped with
> that. It filled up my root disk also...
I think this may have been talking about something else. I don't find any /srv/system- image.ubuntu. com/tmp set. I think
mention of nusakan's root disk filling up in #is-outage, and we run
import-images with TMPDIR=
it's reasonable to have this be the responsibility of the caller rather than
hard-coding such behavior in the code.
> For the moment a cronjob is taking care of removing all these /tmp/tmp*
> files after import-image runs, but ideally import-image should clean up
> its own mess.
In fact, we don't see this behavior here. We do have some tmp files left
behind:
$ du -sh tmp/
2.8G tmp/
$
This is 15 tmp directories that all date to a two-hour window on July 17. I
don't know what was happening on the server then but clearly import-images
was failing abnormally. But as you can see, the vast majority of
import-image runs are cleaning up correctly.
Yes, we should make sure the files are always cleaned up, but we'll need
some idea of how to reproduce this. If you have logs corresponding to the
timestamps of the files left behind, that might be a good place to start.