auth.log fills disk

[Ian Wells]

in a normally running system installed with default configuration, I saw auth.log fill the disk. The quantity of auth logging is highly dependent on what the ovs-agent is doing, and I believe the log rotates once a day, so I would suggest, to reproduce:

- make lots of routers on lots of networks
- leave the system running for a day

ovs-agent will sit there constantly monitoring the vswitch:

Jul 3 14:25:16 control sudo: quantum : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ovs-vsctl --timeout=2 get Interface qr-2c207f2e-cd external_ids
Jul 3 14:25:16 control sudo: pam_unix(sudo:session): session opened for user root by (uid=112)
Jul 3 14:25:16 control sudo: pam_unix(sudo:session): session closed for user root
Jul 3 14:25:16 control sudo: quantum : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ovs-vsctl --timeout=2 get Interface qr-58bb03d6-58 external_ids
Jul 3 14:25:16 control sudo: pam_unix(sudo:session): session opened for user root by (uid=112)
Jul 3 14:25:17 control sudo: pam_unix(sudo:session): session closed for user root
Jul 3 14:25:17 control sudo: quantum : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ovs-vsctl --timeout=2 get Interface qr-6abf5ed7-f7 external_ids
Jul 3 14:25:17 control sudo: pam_unix(sudo:session): session opened for user root by (uid=112)
Jul 3 14:25:17 control sudo: pam_unix(sudo:session): session closed for user root

and fill the disk up.

In my case, the indication was that quantum net-create wouldn't work and quantum agent-list reported all agents were dead (no heartbeats). I believe rabbit was largely inoperable.

[Ian Wells (ijw-ubuntu)]

I would suggest something like:

Defaults:quantum !syslog

- but this would turn off all logging of 'sudo' activity from quantum - it would be better if it were more specific, specifying the actual rootwrap command for no-log treatment.

[Ian Wells (ijw-ubuntu)]

(also, /etc/sudoers.d/quantum-rootwrap is straight out of the package, it seems; we don't reconfigure it, and my changes persist over a puppet rerun.)

[Ian Wells (ijw-ubuntu)]

OK, of the three lines in the syslog, that removes only one - the 'session opened' and 'session closed' lines still need disabling.

[Ian Wells (ijw-ubuntu)]

And, additionally (it removes all PAM session recording, mind - not ideal, though sudo and friends should be logging what people try):

auth,authpriv.notice /var/log/auth.log

either in rsyslog.g/50-default, or in a later file, overriding it.

[Mark T. Voelker (mvoelker)]

What's the burn rate here? E.g. is the right answer simply to rotate the logs so that you can still see when Quantum/OVS are doing things (as I imagine security auditors might well want to know this sort of thing)? Or is disk being consumed to quickly to make that feasible?

[Bolke de Bruin (bolke-de-bruin)]

every 2 seconds per router/tap interface. This is basically creating way too much noise to find anything useful in the auth.log.

-rw-r----- 1 syslog adm 332M Jul 10 12:44 auth.log
-rw-r----- 1 syslog adm 464M Jul 7 06:25 auth.log.1

this is only with 4 routers

[Mark T. Voelker (mvoelker)]

Another thought to add to Ian's: the frequency could also be toned down by changing report_interval in quantum.conf I believe, though that may have other undesirable impacts.

[Chris Ricker (chris-ricker)]

This is a bit fun since the logging is a combination of the PAM module and sudo

Here's my proposed fix:

- change /etc/sudoers.d/quantum_sudoers to include

Defaults:quantum syslog_badpri=err, syslog_goodpri=info

- change /etc/rsyslog.d/50-default.conf auth* priority to

auth.* /var/log/auth.log
authpriv.notice /var/log/auth.log

With that, normal sudo commands still log, as do errored commands from the quantum rootwrap and errors from the PAM session handling

The two things that are not logged in that configuration are

- normal successful quantum rootwrap sudo commands run
- PAM session start / stop for all sudo commands (though the actual command is still logged)

Users in environments which don't mind the noise can simply change authpriv.notice back to authpriv.* in 50-default.conf

Any objections? I'm not thrilled about changing authpriv as there may be other stuff logging at info or debug levels that people care about, but it's probably cleaner than anything else we might do

[Ian Wells (ijw-ubuntu)]

That autopriv setting is very very verbose to my mind so I'm not offended
by changing it.

[Mark T. Voelker (mvoelker)]

I'm also ok with this. Like you I'd prefer not to have to quash the authpriv stuff, but the logspam is bad enough that other such messages are largely getting drowned out anyway so I think it's a reasonable compromise.

[Mark T. Voelker (mvoelker)]

Relevant pull requests:
https://github.com/CiscoSystems/grizzly-manifests/pull/97
https://github.com/CiscoSystems/puppet-coe/pull/2