Excessive logging to /var/log in neutron agents and other containers

[Bjoern]

Even with logrotate running the neutron agents container can fill up:

root@infra01:/opt/rpc-openstack/openstack-ansible/playbooks# ansible -m shell neutron_agent -a "df -h"
infra03_neutron_agents_container-d3e63c0f | success | rc=0 >>
Filesystem Size Used Avail Use% Mounted on
/dev/lxc/infra03_neutron_agents_container-d3e63c0f 4.8G 4.8G 0 100% /
/dev/mapper/lxc-openstack00 246G 67G 179G 28% /var/backup
/dev/mapper/lxc-root00 40G 5.6G 34G 15% /lib/modules
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 13G 104K 13G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 63G 0 63G 0% /run/shm
none 100M 0 100M 0% /run/user

infra02_neutron_agents_container-11c9a132 | success | rc=0 >>
Filesystem Size Used Avail Use% Mounted on
/dev/lxc/infra02_neutron_agents_container-11c9a132 4.8G 4.8G 0 100% /
/dev/mapper/lxc-openstack00 246G 83G 164G 34% /var/backup
/dev/mapper/lxc-root00 40G 5.4G 34G 14% /lib/modules
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 13G 108K 13G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 63G 0 63G 0% /run/shm
none 100M 0 100M 0% /run/user

infra01_neutron_agents_container-49583dcd | success | rc=0 >>
Filesystem Size Used Avail Use% Mounted on
/dev/lxc/infra01_neutron_agents_container-49583dcd 4.8G 4.7G 0 100% /
/dev/mapper/lxc-openstack00 246G 71G 176G 29% /var/backup
/dev/mapper/lxc-root00 40G 5.5G 34G 14% /lib/modules
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 13G 100K 13G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 63G 0 63G 0% /run/shm
none 100M 0 100M 0% /run/user

Which lead to non functioning dhcp agent:

2016-06-01 19:37:28.103 758 TRACE neutron.agent.dhcp.agent OSError: [Errno 28] No space left on device: '/var/lib/neutron/dhcp/acad2a17-564f-40ef-9036-f5bc6a198d71/tmppcqP3U'

at this points I like to discuss treating the containers as actually containers and introduce a logging concept as docker is using.

a) No log file on a local filesystem

b) Logging into bind mounts or c)

c) Better logging to stdout or other destinations:

  1. Logging to stdout and filter the log output into categories (neutron agents, container name etc)
  2. Forwarding to a central rsyslog service (which is installed when logging node configured) or any other service (splunk/logstash/beaver)

d) Bind mounting /var/lib/neutron/dhcp for neutron agents container and possibly other "dynamic data sources" outside of the container

I'm happy to entertain a spec if this is seen helpful

[Bjoern (bjoern-t)]

Additionally it seems that the neutron-ha-tool is now running in debug mode and contributing to large log volume paired with neutron-ns-metadata-proxy logs at /var/log/upstart/neutron-metadata-agent.log

It seems that double logging is happening for neutron-ns-metadata-proxy for example to /var/log/syslog and /var/log/upstart

[Bjoern (bjoern-t)]

Since this issue is similar than the reported one https://bugs.launchpad.net/openstack-ansible/+bug/1491895 I'm going to close the old bug and reiterate my problem here.

[Darren Birkett (darren-birkett)]

Hi Bjoern,

The old bug has some good discussion in it, but a suggestion was made to raise it at the regular meeting to have a real time conversation with all deployers to come to a consensus on the best approach. It's still something worth doing to ensure the correct solution is chosen to fix this issue.

  https://wiki.openstack.org/wiki/Meetings/openstack-ansible

They are held every Thursday at 16:00 UTC.

[Kevin Carter (kevin-carter)]

I think it may be worth our while to examine the log rotate template and see what we can do to ensure we have a proper setup [0]. Maybe the right answer is to update the template to add options with a lower defaults which expose some additional capabilities. In that way a deployer could customize how log rotate works on a per node/deployment basis which would allow for minimal retention in the service container/node and longer retention on the log storage node.

[0] - https://github.com/openstack/openstack-ansible-rsyslog_client/blob/master/templates/os_aggregate_storage.j2

[Bjoern (bjoern-t)]

I'm personally beyond logrotate and have the opinion that whatever we do on this level is just a bandaid. I'm looking for a way not even to accumulate logs in container root disks and do stuff like writing the DHCP agents files out to a bind mount, that alone would have fixed this particular issue, just from the customer impact. I'm not a friend of running logrotate every 5 minutes or so to reduce the likelihood that conntrack fills up the root disk because of a DDOS or whatever.

[Jesse Pretorius (jesse-pretorius)]

There are several short term ways to address this as discussed in the Bug Triage: http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2016-06-07.log.html#t2016-06-07T16:43:08

There is also a longer term solution which needs to be addressed from a design perspective. This should be described in a blueprint/spec with a proposed solution to address it.

[Jesse Pretorius (jesse-pretorius)]

The ideal band-aids for now include:

1 - Resolving the double-logging and log leakage to syslog.
2 - Implement better options for handling log sizes via logrotate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/350697

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/350697
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-rsyslog_client/commit/?id=eea5c79e96a222abf161d0cfa70b19589495e241
Submitter: Jenkins
Branch: master

commit eea5c79e96a222abf161d0cfa70b19589495e241
Author: Andrew Meserole <email address hidden>
Date: Wed Aug 3 16:56:48 2016 +0000

    Prevent syslog leakage

    To prevent leakage to syslog this commit adds local7 to the list
    of excluded facilities in the rsyslog client default configuration
    file. This was done because local7 is used as a facility for
    openstack services to send log messages and by not excluding it
    in the default config file it was sending the log messages to
    syslog and the correct log file for the services.

    Change-Id: Ia938c1266dac51e6f278cadc8272ac98bb270044
    Partial-Bug: 1588051

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/351182

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: liberty
Review: https://review.openstack.org/351184

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351182
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-rsyslog_client/commit/?id=3bcfed8a37a6f09b0ca2b106fb7adfcd72b1e371
Submitter: Jenkins
Branch: stable/mitaka

commit 3bcfed8a37a6f09b0ca2b106fb7adfcd72b1e371
Author: Andrew Meserole <email address hidden>
Date: Wed Aug 3 16:56:48 2016 +0000

    Prevent syslog leakage

    To prevent leakage to syslog this commit adds local7 to the list
    of excluded facilities in the rsyslog client default configuration
    file. This was done because local7 is used as a facility for
    openstack services to send log messages and by not excluding it
    in the default config file it was sending the log messages to
    syslog and the correct log file for the services.

    Change-Id: Ia938c1266dac51e6f278cadc8272ac98bb270044
    Partial-Bug: 1588051
    (cherry picked from commit eea5c79e96a222abf161d0cfa70b19589495e241)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351283

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351299

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351300

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351301

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351302

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351303

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351304

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351305

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351306

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351307

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351308

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351309

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351310

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351311

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351312

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351313

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351314

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351315

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351316

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351317

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351184
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=d79ddc60879e6e636b4511d3895220aa634622a6
Submitter: Jenkins
Branch: liberty

commit d79ddc60879e6e636b4511d3895220aa634622a6
Author: Major Hayden <email address hidden>
Date: Thu Aug 4 08:23:51 2016 -0500

    Prevent syslog leakage

    To prevent leakage to syslog this commit adds local7 to the list
    of excluded facilities in the rsyslog client default configuration
    file. This was done because local7 is used as a facility for
    openstack services to send log messages and by not excluding it
    in the default config file it was sending the log messages to
    syslog and the correct log file for the services.

    Manual backport of Ia938c1266dac51e6f278cadc8272ac98bb270044.

    Partial-Bug: 1588051
    Co-Authored-By: Andrew Meserole <email address hidden>
    Change-Id: I57f2941ccf2d099cdc96798c7902fde8cc2e24ea

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351310
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_magnum/commit/?id=259d5d12e6adf50bfaeb96bb405d59788901cbd7
Submitter: Jenkins
Branch: master

commit 259d5d12e6adf50bfaeb96bb405d59788901cbd7
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:46 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: I851b7648a682ef475f53adb46ecb20ed591c8f0c
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351301
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_ceilometer/commit/?id=1d64ed8927f53a866be6cc00177eab6b4d795bc0
Submitter: Jenkins
Branch: master

commit 1d64ed8927f53a866be6cc00177eab6b4d795bc0
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:04 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: I277bbc3b7b00db8721836cb814548358ca28245c
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351300
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_barbican/commit/?id=01da3d42f160b3401684206a3832351ddf7e358c
Submitter: Jenkins
Branch: master

commit 01da3d42f160b3401684206a3832351ddf7e358c
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:00 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: I22a5a53420f074b64d290e7d19c29343d8556b97
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/351387

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351308
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_ironic/commit/?id=e8b360fa224d83f661ce6a5b1eb6fd4360dde082
Submitter: Jenkins
Branch: master

commit e8b360fa224d83f661ce6a5b1eb6fd4360dde082
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:38 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: I601c764bcb731baeedba784787b721cccfa487e4
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351309
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=866c15301e19ae089fd55f1b7fb35059d93976bd
Submitter: Jenkins
Branch: master

commit 866c15301e19ae089fd55f1b7fb35059d93976bd
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:42 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: Ia65b73351f2cbca5b14ae124789a8e33b53cfd39
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/351390

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351856
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=1432dea62ad4c63e69001e9a55cf543e7fb43012
Submitter: Jenkins
Branch: stable/mitaka

commit 1432dea62ad4c63e69001e9a55cf543e7fb43012
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:42 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: Ia65b73351f2cbca5b14ae124789a8e33b53cfd39
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit 866c15301e19ae089fd55f1b7fb35059d93976bd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351854
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_heat/commit/?id=dcff3d073520961d5e0e692aad0d5636254a3a0b
Submitter: Jenkins
Branch: stable/mitaka

commit dcff3d073520961d5e0e692aad0d5636254a3a0b
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:30 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: I00ae0eba37e0f5217461acaecdbec03edae3c43b
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit ffce76c2facf893330724f758b7e48cb8db33eea)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351855
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_neutron/commit/?id=5a53856cb33b2d8b367b76416ffa570471ca6897
Submitter: Jenkins
Branch: stable/mitaka

commit 5a53856cb33b2d8b367b76416ffa570471ca6897
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:51 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: I8b8c4172a8bed54c507ebe6d54f9b3db6669afe4
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit a597bf073db8448a112d3047e9b7796371f2292b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351852
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_glance/commit/?id=ec9ae0917183b644c61a1fb6d74b4fdf34817e30
Submitter: Jenkins
Branch: stable/mitaka

commit ec9ae0917183b644c61a1fb6d74b4fdf34817e30
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:21 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: I40e18d531d7551c6a460449dd17116abd68187b2
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit 7f4c2acbac2ce9196561c598967881acfbcb7044)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351390
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_ironic/commit/?id=03443e4d9b1dcb939f34c521a99b17022ec71470
Submitter: Jenkins
Branch: stable/mitaka

commit 03443e4d9b1dcb939f34c521a99b17022ec71470
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:38 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: I601c764bcb731baeedba784787b721cccfa487e4
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit e8b360fa224d83f661ce6a5b1eb6fd4360dde082)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351312
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=ba9b97e05a13f369d91c6e4be334a457f5f94771
Submitter: Jenkins
Branch: master

commit ba9b97e05a13f369d91c6e4be334a457f5f94771
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:55 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: I1974b9ad4080aa4253de9e3112f0973d44aa23f8
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/352461

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/352461
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=bd83209c1948eefb079f3fc1b8775094d89c510f
Submitter: Jenkins
Branch: stable/mitaka

commit bd83209c1948eefb079f3fc1b8775094d89c510f
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:55 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: I1974b9ad4080aa4253de9e3112f0973d44aa23f8
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351302
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_cinder/commit/?id=532025c21dd543745123c5b1120e5a41b2af2988
Submitter: Jenkins
Branch: master

commit 532025c21dd543745123c5b1120e5a41b2af2988
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:08 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: I10d73991d566d0ad0f6a91cf11cf835a8299e387
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/354080

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/354080
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_cinder/commit/?id=77f138e8894d161ec487af1e3447cd27e3b161d9
Submitter: Jenkins
Branch: stable/mitaka

commit 77f138e8894d161ec487af1e3447cd27e3b161d9
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 12:36:08 2016 -0500

    Disable stderr logging

    OSLO logging currently defaults the 'use_stderr' option to True
    which results duplicate logs in service daemon logs for both
    upstart and systemd. To correct this issue the use_stderr
    option has been set to false.

    Change-Id: I10d73991d566d0ad0f6a91cf11cf835a8299e387
    Closes-Bug: 1588051
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit 532025c21dd543745123c5b1120e5a41b2af2988)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: liberty
Review: https://review.openstack.org/356107

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/356107
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=4e193a7d64c5e76d166e2cbd2d6d69b308eb0b8c
Submitter: Jenkins
Branch: liberty

commit 4e193a7d64c5e76d166e2cbd2d6d69b308eb0b8c
Author: Ian Cordasco <email address hidden>
Date: Tue Aug 16 14:21:38 2016 -0500

    Set use_stderr to False for all services

    oslo.log has a setting to log to stderr as well as a log file and
    defaults it to True. This causes OpenStack Ansible to log each service's
    logs twice: once to /var/log/{{ service }}/{{ binary-name }}.log and
    once to /var/log/upstart/{{ binary-name }}.log because upstart views
    each log event on stderr as a priority. Defaulting this to False means
    we reduce the duplication of log events on each container and that the
    upstart logs are actually informative rather than confusing.

    Change-Id: I9d2b2ca74fa75b7ec38de4b6fe01fdf73af03fe2
    Closes-bug: 1588051

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_keystone 13.3.2 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_aodh 13.3.2 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_ceilometer 13.3.2 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_cinder 13.3.2 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_glance 13.3.2 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_heat 13.3.2 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_ironic 13.3.2 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_neutron 13.3.2 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_nova 13.3.2 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_swift 13.3.2 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_tempest 13.3.2 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351283
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=15526473b0634a6b0cf63b14b2f01daa6f792dd6
Submitter: Jenkins
Branch: master

commit 15526473b0634a6b0cf63b14b2f01daa6f792dd6
Author: Ian Cordasco <email address hidden>
Date: Thu Aug 4 11:35:19 2016 -0500

    Default use_stderr to False

    While defaulting use_stderr to True might be a useful development
    default, it is not useful for operators. Operators using systemd,
    upstart, or any of these kinds of services, will find that if they
    specify --log-file they will have two copies of the same logs on their
    disk:

    - one in the service log (e.g., /var/log/upstart)
    - one in the log file they specify with --log-file

    Instead of operators having to disable stderr log in every
    configuration, we should just default it to False here.

    Closes-bug #1588051

    Change-Id: I85f8670d98d03101888fb7bb0c89eb809719b4e8

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 12.2.2 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351188
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=992e616046a737357183394808ea5a0c48a49065
Submitter: Jenkins
Branch: master

commit 992e616046a737357183394808ea5a0c48a49065
Author: Kevin Carter <email address hidden>
Date: Thu Aug 4 08:35:41 2016 -0500

    Implement container bind mount for all logs

    All of the log directories for all containers will now be bind mounted to the
    host. This change ensures that containers are not running into an issue with
    Full file systems due to logs which is common when a container is backed by
    a blocked device (lvm, zfs, btrfs).

    Closes-Bug: #1588051
    Change-Id: I25a481c0409f1a45494a8668f00c5393672e853c
    Signed-off-by: Kevin Carter <email address hidden>

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 14.0.0.0b3 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_keystone 14.0.0.0b3 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_aodh 14.0.0.0b3 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_ceilometer 14.0.0.0b3 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_cinder 14.0.0.0b3 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_glance 14.0.0.0b3 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_gnocchi 14.0.0.0b3 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_heat 14.0.0.0b3 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_ironic 14.0.0.0b3 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_neutron 14.0.0.0b3 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_nova 14.0.0.0b3 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_swift 14.0.0.0b3 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_tempest 14.0.0.0b3 development milestone.