Turn off log file logging in conntrackd

[Bjoern]

Please furn off log file logging in conntrackd completely.
Currently it does not use syslog and writes it own log file to /var/log
Please set LogFile to off as a default override.

[Kevin Carter (kevin-carter)]

The logging for contractd is an issue however Im not sure we want to disable logging all together. Maybe we should change the rotate rules?

[Kevin Carter (kevin-carter)]

Bump. is this still a problem? Is rotating more often a solution?

[Jimmy McCrory (jimmy-mccrory)]

I only manage Juno deployments right now, but it's definitely a problem there.
These logs don't seem very useful, grow quickly, and there might be a bug with the package itself since the config says the default logging should be off.

Either rotating more often or disabling completely would help though.

[Darren Birkett (darren-birkett)]

conntrackd has the ability to create 2 kinds of logs:

/var/log/conntrackd.log
/var/log/conntrackd-stats.log

In the default package config, the first is off and the second is on, and this is what I observe on my deployments:

juno:
root@aio1_neutron_agents_container-28d6792d:~# ll /var/log/conntrackd*
-rw------- 1 root root 215460 Jul 7 12:23 /var/log/conntrackd-stats.log
-rw------- 1 root root 573696 Jul 5 06:23 /var/log/conntrackd-stats.log.1
-rw------- 1 root root 726426 Jun 29 06:23 /var/log/conntrackd-stats.log.2

kilo:
root@aio1_neutron_agents_container-3b2d882a:~# ll /var/log/conntrackd*
-rw------- 1 root root 363216 Jul 7 12:26 /var/log/conntrackd-stats.log
-rw------- 1 root root 1130436 Jul 5 06:23 /var/log/conntrackd-stats.log.1
-rw------- 1 root root 1472614 Jun 28 06:23 /var/log/conntrackd-stats.log.2

Could we get some more info:

- exactly what logfiles are we talking about
- which containers/hosts?
- are they ever useful? or is disabling them completely the best option?

[Jimmy McCrory (jimmy-mccrory)]

conntrackd-stats logs
neutron-agents and compute hosts
haven't found them personally useful

This is from a ~140 compute node Juno deployment, other environments' containers were also nearly full.
We disable conntrackd-stats logging after truncating/removing these logs.

ansible neutron_agent -m shell -a 'ls -lh /var/log/conntrack*'

infra1_neutron_agents_container-8ab495ee | success | rc=0 >>
-rw------- 1 root root 1010M Jul 6 13:34 /var/log/conntrackd-stats.log
-rw------- 1 root root 3.0G Jul 5 06:25 /var/log/conntrackd-stats.log.1
-rw------- 1 root root 963K Jun 29 06:24 /var/log/conntrackd-stats.log.2

infra2_neutron_agents_container-34202426 | success | rc=0 >>
-rw------- 1 root root 3.9G Jul 6 13:34 /var/log/conntrackd-stats.log
-rw------- 1 root root 3.2M Jun 28 06:25 /var/log/conntrackd-stats.log.1
-rw------- 1 root root 6.3M Jun 21 06:24 /var/log/conntrackd-stats.log.2

infra3_neutron_agents_container-bd50a84b | success | rc=0 >>
-rw------- 1 root root 706M Jul 6 13:34 /var/log/conntrackd-stats.log
-rw------- 1 root root 3.2G Jul 5 06:25 /var/log/conntrackd-stats.log.1
-rw------- 1 root root 3.5M Jun 29 06:24 /var/log/conntrackd-stats.log.2

[Darren Birkett (darren-birkett)]

Thanks Jimmy,

I wonder if we need to have conntrackd installed at all. It's prime use is to be able to synchronise state between linux firewalls, in an HA firewall environment (clearly we are not using it for that in our deployments). A secondary use is to gather, and log, connection statistics. It seems that if we generally find the stats logs useless, there's not much point in conntrackd running.

Note, removing conntrackd does not unload the kernel conntrack modules or disable the netfilter conntrack framework. It simply means we are not gathering ostensibly useless logging. The kernel conntrack flows can still be interrogated with the 'conntrack' userspace cli tool, for real time debugging.

Soliciting further opinion.

[Bjoern (bjoern-t)]

I agree turning it off is beneficial, as I already said we are not taking any advantage out of the log file so why generating them in the first place

[Evan Callicoat (diopter)]

Apparently conntrackd is part of the stated design of Neutron's L3 HA implementation. However, it seems that this does not in fact exist in current Neutron master code. In addition, even if it was implemented and being used by Neutron, that definitely does not involve running the service with the stock package config files outside of router namespaces in agent containers, and double definitely does not involve constantly logging the copious conntrack state changes.

My suggestion is to remove the package installation entirely for the time being, and everyone will be a lot happier for it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: juno
Review: https://review.openstack.org/199574

[Darren Birkett (darren-birkett)]

Fix proposed to branch: master
Review: https://review.openstack.org/199572

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/199574
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=c9eb30451b613c51556cb4f4d602db5fd26eadd2
Submitter: Jenkins
Branch: juno

commit c9eb30451b613c51556cb4f4d602db5fd26eadd2
Author: Darren Birkett <email address hidden>
Date: Wed Jul 8 12:10:29 2015 +0000

    remove conntrackd package

    As per bug discussion, conntrackd is actually not needed for netfilter
    framework, and given that users do not appear to find the logs useful
    (the only reason we're using conntrackd), let's just remove it.

    This commit also defines a variable container_remove_packages so that
    the list of packages to remove from containers can be added to in the
    future.

    NOTE: removing conntrackd does not unload the kernel conntrack modules
    or disable the netfilter conntrack framework. It simply means we are not
    gathering ostensibly useless logging. The kernel conntrack flows can
    still be interrogated with the 'conntrack' userspace cli tool, for real
    time debugging.

    Change-Id: Ic74e65a6fe27060dc94bfc2f250cd53fb153c7c8
    Closes-Bug: 1457196

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/199572
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=0da48344c6bdff988473da1aa7e9e0f6ef724680
Submitter: Jenkins
Branch: master

commit 0da48344c6bdff988473da1aa7e9e0f6ef724680
Author: Darren Birkett <email address hidden>
Date: Wed Jul 8 13:42:01 2015 +0000

    remove conntrackd package

    As per bug discussion, conntrackd is actually not needed for netfilter
    framework, and given that users do not appear to find the logs useful
    (the only reason we're using conntrackd), let's just remove it.

    This commit also defines a variable container_remove_packages so that
    the list of packages to remove from containers can be added to in the
    future.

    NOTE: removing conntrackd does not unload the kernel conntrack modules
    or disable the netfilter conntrack framework. It simply means we are not
    gathering ostensibly useless logging. The kernel conntrack flows can
    still be interrogated with the 'conntrack' userspace cli tool, for real
    time debugging.

    Change-Id: Ic74e65a6fe27060dc94bfc2f250cd53fb153c7c8
    Closes-Bug: 1457196

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/207441
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=811a099205354240e8a462f8978175a166c2358c
Submitter: Jenkins
Branch: kilo

commit 811a099205354240e8a462f8978175a166c2358c
Author: Darren Birkett <email address hidden>
Date: Wed Jul 8 13:42:01 2015 +0000

    remove conntrackd package

    As per bug discussion, conntrackd is actually not needed for netfilter
    framework, and given that users do not appear to find the logs useful
    (the only reason we're using conntrackd), let's just remove it.

    This commit also defines a variable container_remove_packages so that
    the list of packages to remove from containers can be added to in the
    future.

    NOTE: removing conntrackd does not unload the kernel conntrack modules
    or disable the netfilter conntrack framework. It simply means we are not
    gathering ostensibly useless logging. The kernel conntrack flows can
    still be interrogated with the 'conntrack' userspace cli tool, for real
    time debugging.

    Change-Id: Ic74e65a6fe27060dc94bfc2f250cd53fb153c7c8
    Closes-Bug: 1457196
    (cherry picked from commit 0da48344c6bdff988473da1aa7e9e0f6ef724680)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.11 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 11.2.12 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.14 release.