Excessive logging to /var/log in neutron agents and other containers
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack-Ansible |
Fix Released
|
Low
|
Kevin Carter | ||
Liberty |
Fix Released
|
Medium
|
Ian Cordasco | ||
Mitaka |
Fix Released
|
Medium
|
Ian Cordasco | ||
Trunk |
Fix Released
|
Low
|
Kevin Carter | ||
oslo.log |
Fix Released
|
Undecided
|
ChangBo Guo(gcb) |
Bug Description
Even with logrotate running the neutron agents container can fill up:
root@infra01:
infra03_
Filesystem Size Used Avail Use% Mounted on
/dev/lxc/
/dev/mapper/
/dev/mapper/
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 13G 104K 13G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 63G 0 63G 0% /run/shm
none 100M 0 100M 0% /run/user
infra02_
Filesystem Size Used Avail Use% Mounted on
/dev/lxc/
/dev/mapper/
/dev/mapper/
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 13G 108K 13G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 63G 0 63G 0% /run/shm
none 100M 0 100M 0% /run/user
infra01_
Filesystem Size Used Avail Use% Mounted on
/dev/lxc/
/dev/mapper/
/dev/mapper/
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 13G 100K 13G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 63G 0 63G 0% /run/shm
none 100M 0 100M 0% /run/user
Which lead to non functioning dhcp agent:
2016-06-01 19:37:28.103 758 TRACE neutron.
at this points I like to discuss treating the containers as actually containers and introduce a logging concept as docker is using.
a) No log file on a local filesystem
b) Logging into bind mounts or c)
c) Better logging to stdout or other destinations:
1. Logging to stdout and filter the log output into categories (neutron agents, container name etc)
2. Forwarding to a central rsyslog service (which is installed when logging node configured) or any other service (splunk/
d) Bind mounting /var/lib/
I'm happy to entertain a spec if this is seen helpful
Changed in openstack-ansible: | |
assignee: | nobody → Andrew Meserole (andrew-meserole) |
Changed in oslo.log: | |
status: | New → Triaged |
Changed in oslo.log: | |
assignee: | Ian Cordasco (icordasc) → ChangBo Guo(gcb) (glongwave) |
Additionally it seems that the neutron-ha-tool is now running in debug mode and contributing to large log volume paired with neutron- ns-metadata- proxy logs at /var/log/ upstart/ neutron- metadata- agent.log
It seems that double logging is happening for neutron- ns-metadata- proxy for example to /var/log/syslog and /var/log/upstart