excessive logs in /var/log/eucalyptus

[Scott Moser]

<kirkland> ubuntu@cluster:/var/log/eucalyptus$ du -sh .
<kirkland> 191M .
<kirkland> 1 day old
<kirkland> yes
<kirkland> please file a bug on it
<kirkland> i think their is a cronjob
<kirkland> we need a /etc/logrotate.d/eucalyptus

smoser@jimbo:~/uec$ uptime
 16:26:36 up 2:31, 3 users, load average: 0.12, 0.12, 0.09
smoser@jimbo:~/uec$ du -hs /var/log/eucalyptus/
23M /var/log/eucalyptus/

The above shows fairly big need for log rotate. At my current pace, I'll have ~ 200M per day. This is on a 1 node cluster with so far a single instance. Anything actually busy would possibly be heavier.

[Thierry Carrez (ttx)]

It also needs saner default logging levels. Everything is at "Debug" level by default and there is a fair amount of redundancy between cloud-output.log and cloud-debug.log. In inactive periods, lines are being added to cc.log, nc.log and cloud-output.log every 8 seconds or so. While this is perfectly acceptable in debug mode, that seems very noisy in production mode.

[Dustin Kirkland  (kirkland)]

Dan Nurmi says that Eucalyptus rotates its own logs, so they won't grow indefinitely.

I filed this bug because I saw my logs getting huge, but no jobs in /etc/logrotate.d.

In this case, the rotation aspect of this bug is already solved. But the log verbosity should be tuned to something reasonable for users, rather than developers.

:-Dustin

[Thierry Carrez (ttx)]

Setting LOGLEVEL="ERROR" in /etc/eucalyptus/eucalyptus.conf doesn't seem to change the verbosity of cloud-output.log or cloud-error.log...

[Matt Zimmerman (mdz)]

Can someone provide a pointer to the relevant code in Eucalyptus which does the log rotation? We need to confirm if this works as expected.

[Daniel Nurmi (nurmi)]

the java (cloud, walrus, sc) components' logging uses log4j. the defaults can be found in source under:

clc/modules/core/src/main/resources/log4j.xml

where you'll find the following, which control the file size and the max number of files of this size that can exist

       <param name="MaxFileSize" value="10MB"/>
       <param name="MaxBackupIndex" value="10"/>

for the C components (cc, nc), the logfiles are controlled inside the logprintfl() function, where you'll find logic for handling the roll-over. by default, there can be six files (ex: cc.log, cc.log{0-5}), each of which at most 32MB.

the axis2c.log is rolled once by axis2c itself (axis2c.log and axis2c.log.old), and its size can be set through httpd.conf (128MB by default)

Axis2MaxLogFileSize 128

the httpd-{cc,nc}-error_log (set in httpd.conf, again) is not rolled, but tends to grow very slowly.

finally, i'm not sure where the jetty log rolling policy is set, but will find out and comment here.

[Thierry Carrez (ttx)]

Log rotation appears to work as expected, though I didn't run it over that many days:

$ ls -s /var/log/eucalyptus/
total 45200
 7996 axis2c.log
 5824 cc.log
    4 cc-registration.log
 8036 cloud-debug.log
10248 cloud-debug.log.1
10248 cloud-debug.log.2
  284 cloud-error.log
 2480 cloud-output.log
   12 httpd-cc_error_log
    4 jetty-request-2009_10_21.log
    4 jetty-request-2009_10_22.log
    4 sc-registration.log
    4 sc-stats.log
    4 walrus-registration.log
   48 walrus-stats.log

Closing as invalid, opening a separate bug for the issue mentioned in comment 3.