logstash: almost all logs aren't properly processed

[Jesse Pretorius]

It would appear that most logs are either dropped or not being properly processed by logstash. When all the filters are removed then the logs show properly in Kibana when the correct query is used.

It may be best to go back to basics in the log processing and refactor it to ensure that absolutely all logs are pushed into Elasticsearch and shown in Kibana.

Some inspiration:
https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/templates/logstash/agent.conf.erb
https://github.com/godaddy/openstack-logstash

[Bjoern (bjoern-t)]

This is actually a known issue ever since and based on the github issue #530.
Here the based launchpad issue: https://bugs.launchpad.net/openstack-ansible/+bug/1399410

[Jesse Pretorius (jesse-pretorius)]

You may note that the CPU utilization issue has been resolved by turning logstash-web off. This is an entirely different issue - logstash is quite literally dropping messages after processing through them. The filters are dumping them.

[Jesse Pretorius (jesse-pretorius)]

Fix proposed: https://review.openstack.org/145195

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/145195
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=2231094cfd19a47469dab42fbcab59cbb271eb95
Submitter: Jenkins
Branch: master

commit 2231094cfd19a47469dab42fbcab59cbb271eb95
Author: Jesse Pretorius <email address hidden>
Date: Tue Jan 6 10:36:19 2015 +0000

    Implement logstash filter changes to improve log processing

    This patch implements changes to the logstash filter configurations in
    order to improve log processing such that all logs are tagged and passed
    through to elasticsearch properly. It implements a change in the initial
    'rsyslog' filter logic and several tweaks in the follow-up filters,
    resulting in no more grok parse failures.

    This patch also includes log parsing for swift, neutron and the mysql
    slow query log.

    Co-Authored-By: Steve Lewis <email address hidden>
    Change-Id: Id6180b080f11fada00386e39c64f8d7338f8c94d
    Closes-Bug: #1403890
    Closes-Bug: #1399371
    Closes-Bug: #1399367

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/148623

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/148623
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=f932317df85b01de59222cedd25909a56952cdba
Submitter: Jenkins
Branch: master

commit f932317df85b01de59222cedd25909a56952cdba
Author: Jesse Pretorius <email address hidden>
Date: Fri Jan 2 16:52:13 2015 +0000

    Ensure that permissions are permissible enough for the rsyslog container

    The rsyslog container is unable to read the log files produced by the
    other containers due to permissions issues and inconsistant user:group
    ownership and user:group presence between containers.

    The simplest way of dealing with this is to ensure that the log
    directory allows read and traversal rights for all users.

    This patch ensures that the log directory is always created by the
    container_common role for containers.

    This patch also add the processing of swift logs to logstash. The
    template was always there, but had no task which deployed it.

    Change-Id: I6ec12b2f2302e0cffc2a36096d38cb85e4b3ab04
    Closes-Bug: #1407633
    Related-Bug: #1403890

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: juno
Review: https://review.openstack.org/150361

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: juno
Review: https://review.openstack.org/150363

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: icehouse
Review: https://review.openstack.org/150561

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/150561
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=5a44cc7433e608814d2f73abc36092f46123024d
Submitter: Jenkins
Branch: icehouse

commit 5a44cc7433e608814d2f73abc36092f46123024d
Author: Jesse Pretorius <email address hidden>
Date: Tue Jan 6 10:36:19 2015 +0000

    Implement logstash filter changes to improve log processing

    This patch implements changes to the logstash filter configurations in
    order to improve log processing such that all logs are tagged and passed
    through to elasticsearch properly. It implements a change in the initial
    'rsyslog' filter logic and several tweaks in the follow-up filters,
    resulting in no more grok parse failures.

    This patch also includes log parsing for swift, neutron and the mysql
    slow query log.

    Co-Authored-By: Steve Lewis <email address hidden>
    Change-Id: Id6180b080f11fada00386e39c64f8d7338f8c94d
    Closes-Bug: #1403890
    Closes-Bug: #1399371
    Closes-Bug: #1399367
    (cherry picked from commit 2231094cfd19a47469dab42fbcab59cbb271eb95)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/150361
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=86327da7d4d8d70c740aa57c1595f8f2a04c366c
Submitter: Jenkins
Branch: juno

commit 86327da7d4d8d70c740aa57c1595f8f2a04c366c
Author: Jesse Pretorius <email address hidden>
Date: Tue Jan 6 10:36:19 2015 +0000

    Implement logstash filter changes to improve log processing

    This patch implements changes to the logstash filter configurations in
    order to improve log processing such that all logs are tagged and passed
    through to elasticsearch properly. It implements a change in the initial
    'rsyslog' filter logic and several tweaks in the follow-up filters,
    resulting in no more grok parse failures.

    This patch also includes log parsing for swift, neutron and the mysql
    slow query log.

    Co-Authored-By: Steve Lewis <email address hidden>
    Change-Id: Id6180b080f11fada00386e39c64f8d7338f8c94d
    Closes-Bug: #1403890
    Closes-Bug: #1399371
    Closes-Bug: #1399367
    (cherry picked from commit 2231094cfd19a47469dab42fbcab59cbb271eb95)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/150363
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=831426addc9b378008b0334eb51dc44e97efe48f
Submitter: Jenkins
Branch: juno

commit 831426addc9b378008b0334eb51dc44e97efe48f
Author: Jesse Pretorius <email address hidden>
Date: Fri Jan 2 16:52:13 2015 +0000

    Ensure that permissions are permissible enough for the rsyslog container

    The rsyslog container is unable to read the log files produced by the
    other containers due to permissions issues and inconsistant user:group
    ownership and user:group presence between containers.

    The simplest way of dealing with this is to ensure that the log
    directory allows read and traversal rights for all users.

    This patch ensures that the log directory is always created by the
    container_common role for containers.

    This patch also add the processing of swift logs to logstash. The
    template was always there, but had no task which deployed it.

    Change-Id: I6ec12b2f2302e0cffc2a36096d38cb85e4b3ab04
    Closes-Bug: #1407633
    Related-Bug: #1403890
    (cherry picked from commit f932317df85b01de59222cedd25909a56952cdba)