all-machines.log is oversized on juju node

[Kurt]

It appears #1195223 is not fixed in juju 1.12. all-machines.log is growing unbridled. It fills up the entire file system. It is 12G total in size:

ubuntu@tada6:/var/log/juju$ ls -al
total 12748880
drwxr-xr-x 2 root root 4096 Aug 29 17:14 .
drwxr-xr-x 13 root root 4096 Aug 29 20:14 ..
-rw-r----- 1 syslog adm 13047238656 Aug 29 20:55 all-machines.log
-rw-r--r-- 1 root root 5338 Aug 29 17:14 juju-gui.log
-rw-r--r-- 1 root root 4587971 Aug 29 20:55 machine-0.log
-rw-r--r-- 1 root root 2992193 Aug 29 20:55 unit-juju-gui-0.log

Can we either:
1. Rotate and compress the log upon certain size
2. Delete the log when it reaches a certain percentage of total files system size
3. Figure out what is filling it up and reduce that logging by default

[Kurt (burnbrighter)]

tada6:2ubuntu@tada6:/var/log/juju$ ls -al /etc/rsyslog*
-rw-r--r-- 1 root root 1263 Mar 30 2012 /etc/rsyslog.conf

/etc/rsyslog.d:
total 28
drwxr-xr-x 2 root root 4096 Aug 29 17:13 .
drwxr-xr-x 94 root root 4096 Aug 29 20:14 ..
-rw-r--r-- 1 root root 311 Mar 17 2012 20-ufw.conf
-rw-r--r-- 1 root root 252 Aug 16 03:17 21-cloudinit.conf
-rw-r--r-- 1 root root 709 Aug 29 16:55 25-juju.conf
-rw-r--r-- 1 root root 337 Aug 29 17:13 26-juju-unit-juju-gui-0.conf
-rw-r--r-- 1 root root 1655 Mar 30 2012 50-default.conf

ubuntu@tada6:/var/log/juju$ cat /etc/rsyslog.d/50-default.conf
# Default rules for rsyslog.
#
# For more information see rsyslog.conf(5) and /etc/rsyslog.conf

#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
#daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
#lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
#user.* -/var/log/user.log

#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
#mail.info -/var/log/mail.info
#mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err

#
# Logging for INN news system.
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice

#
# Some "catch-all" log files.
#
#*.=debug;\
# auth,authpriv.none;\
# news.none;mail.none -/var/log/debug
#*.=info;*.=notice;*.=warn;\
# auth,authpriv.none;\
# cron,daemon.none;\
# mail,news.none -/var/log/messages

#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*

#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8

# The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# you must invoke `xconsole' with the `-file' option:
#
# $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# busy site..
#
daemon.*;mail.*;\
 news.err;\
 *.=debug;*.=info;\
 *.=notice;*.=warn |/dev/xconsole

[Kurt (burnbrighter)]

After sending the logging to /dev/null, the cpu is now spinning out of control

ubuntu@tada6:~$ top

top - 21:36:44 up 10 min, 1 user, load average: 1.53, 1.37, 0.77
Tasks: 78 total, 1 running, 77 sleeping, 0 stopped, 0 zombie
Cpu(s): 39.5%us, 45.8%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 14.6%si, 0.0%st
Mem: 2044512k total, 656240k used, 1388272k free, 63128k buffers
Swap: 2094076k total, 0k used, 2094076k free, 400400k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  950 syslog 20 0 391m 6908 1268 S 99.1 0.3 9:47.22 rsyslogd <------
 1044 root 20 0 383m 49m 23m S 0.3 2.5 0:02.29 mongod
 1950 ubuntu 20 0 79616 1640 808 S 0.3 0.1 0:00.16 sshd
    1 root 20 0 24440 2432 1372 S 0.0 0.1 0:01.09 init
    2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
    3 root 20 0 0 0 0 S 0.0 0.0 0:00.34 ksoftirqd/0
    4 root 20 0 0 0 0 S 0.0 0.0 0:00.44 kworker/0:0
    6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
    7 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
    8 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cpuset

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2013-08-30 5:03, Ian Booth wrote:
> ** Changed in: juju-core Importance: Undecided => Critical
>
> ** Changed in: juju-core Status: New => Triaged
>

The report is about juju-1.12. I thought we fixed both the unbridled
growth and we log-rotate it in 1.13.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlIiP2QACgkQJdeBCYSNAANpAwCcCBctcwgrE6zVzHOR/8+dVpdz
bD4AoK2DvaycnXpyirRMCQ4GObCpJaT/
=zGCL
-----END PGP SIGNATURE-----