DHCP server is too chatty

[Evan Broder]

The DHCP server dominates almost 90% of our hosts' syslogs:

root@aperture-science:~# less /var/log/syslog | grep -iv python | wc -l
722
root@aperture-science:~# less /var/log/syslog | grep python | wc -l
6239

And it's relatively useless spew, too:

Oct 16 06:52:49 aperture-science python: Get DHCPDISCOVER packet from ff:12:34:56:78:9a
Oct 16 06:52:49 aperture-science python: dhcp_backend : Discover
Oct 16 06:52:49 aperture-science python: dhcp_backend : Discover result = 1.2.3.4

Particularly useless, because most of the traffic is from our monitoring service, which sends 2 DHCPDISCOVERs every 5 minutes.

Currently all of our syslog messages are at level DEBUG or INFO. Both of those seem like they should be filtered out for a production service. If there's anything that matters, it should be logged at a higher priority level, and we should filter out the spew.

[Quentin Smith (quentin-mit)]

It's reasonable and common for DHCP servers to log once for each DHCP packet and response; e.g. ISC's dhcpd will log "dhcpd: DHCPREQUEST for 18.238.6.84 from 00:1d:e0:8c:ce:01 via eth0" for a request, or "dhcpd: DHCPOFFER on 192.168.0.49 to 00:03:93:20:81:ba via eth1". We should still perform logging of this type; probably each of the individual logging calls in different functions should be coalesced into one line that logs all the relevant information.

[Greg Price (gregprice)]

Reducing everything to one line would cut everything down by a factor of four, but that'd still make it two-thirds of our syslog by Evan's count, which is still annoying for figuring out what's going on with other parts of the system.

I'm not convinced we need these logs in production. But if we want them, perhaps we can send them somewhere else, perhaps by using Python's logging module rather than the syslog module. (And probably also condense them as Quentin suggests. Easier to grep, etc, that way.)