importds are leaving tmp files in /tmp

[Steve McInerney]

    pear:~$ ls -laU /tmp | wc -l
        359746
    russkaya:~$ ls -laU /tmp | wc -l
        300302
    galapagos:~$ ls -laU /tmp | wc -l
        201800
    neumayer:~$ ls -laU /tmp | wc -l
        8321

Numbers as at 2011-07-13 ~08:15 UTC.

The files are all of tmp*.idx or .pack.
We seem to be increasing by several thousand per day.

neumayer is an interesting difference. A day later it's up to 11,000 tho.

[Gary Poster (gary)]

Is this a regression, or something that should be escalated from an operational perspective? If so, critical might be more apt.

[Steve McInerney (spm)]

This just triggered a disk space alert again on pear.

Have put in an evil hack:
5 * * * * find /tmp -maxdepth 1 -type f \( -name 'tmp*.pack' -o -name 'tmp*.idx' \) -mtime +5 -delete

But this is mindless. If something is still using one of those tmp files ....

[Steve McInerney (spm)]

Have had to reduce this to a 3 day cleanup.
We were holding ~ 55Gb in non cleaned up junk in /tmp
Now down to ~ 35Gb.

[Haw Loeung (hloeung)]

strawberry had 2.7G of *.idx and *.pack files in /tmp

[Haw Loeung (hloeung)]

-rw------- 1 importd importd 0 Aug 20 10:28 tmpUke9if.pack
-rw------- 1 importd importd 0 Aug 20 10:28 tmpJ9UIMM.pack
-rw------- 1 importd importd 0 Aug 20 10:28 tmpNC4v7j.pack
-rw------- 1 importd importd 0 Aug 20 10:25 tmpMF2IUk.pack
-rwxr-xr-x 1 importd importd 2052 Aug 20 10:25 tmppiVwPx.idx
-rw------- 1 importd importd 65610 Aug 20 10:25 tmppiVwPx.pack
-rw------- 1 importd importd 0 Aug 20 10:24 tmpS7rhKf.pack
-rw------- 1 importd importd 0 Aug 20 10:24 tmpTvMBRp.pack
-rw------- 1 importd importd 0 Aug 20 10:22 tmp1QnrnJ.pack
-rw------- 1 importd importd 0 Aug 20 10:22 tmp6z1lSi.pack
-rw------- 1 importd importd 86405 Aug 20 10:20 tmphC2tcG.pack
hloeung@neumayer:/tmp$ du -sh
du: cannot read directory `./gpg-pAwBzR': Permission denied
du: cannot read directory `./gpg-VeHZVj': Permission denied
329G .
hloeung@neumayer:/tmp$

Using spm's evil hack brought it down to 15G. Applied evil hack from pear to neumayer and other importds (russkaya and galapagos).

[Haw Loeung (hloeung)]

importd@galapagos:/tmp$ du -sh
340G .
importd@galapagos:/tmp$ find /tmp -maxdepth 1 -type f \( -name 'tmp*.pack' -o -name 'tmp*.idx' \) -mtime +1 -delete
importd@galapagos:/tmp$ du -sh
5.9G .
importd@galapagos:/tmp$

importd@russkaya:/tmp$ du -sh
315G .
importd@russkaya:/tmp$ find /tmp -maxdepth 1 -type f \( -name 'tmp*.pack' -o -name 'tmp*.idx' \) -mtime +1 -delete
importd@russkaya:/tmp$ du -sh
7.3G .
importd@russkaya:/tmp$