logstash has empty "%{logmessage}" messages vs. actual content for many files

[Patrick Crews]

There are a number of files in logstash where the message content is: '%{logmessage}' vs. what is actually in the file.
This is true for several files.
Some examples:
libvirtd.txt
host_info.txt
os-collect-config.txt

http://logstash.openstack.org/#eyJzZWFyY2giOiIgbWVzc2FnZTpcIiV7bG9nbWVzc2FnZX1cIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiIsInN0YW1wIjoxNDA2NjgxMjg4NDA2fQ==

Initially thought it was tied to just tripleo logs, but it appears to affect other jobs as well.
Likely needing to update this file: modules/openstack_project/templates/logstash/indexer.conf.erb
https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/logstash/indexer.conf.erb

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/111385

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/111385
Committed: https://git.openstack.org/cgit/openstack-infra/config/commit/?id=3ad20ce40a3d373bc080daeefb48fdff7293ffd9
Submitter: Jenkins
Branch: master

commit 3ad20ce40a3d373bc080daeefb48fdff7293ffd9
Author: pcrews <email address hidden>
Date: Fri Aug 1 13:31:34 2014 -0700

    Altering indexer config to properly match on libvirtd files

    The tags associated with libvirtd.txt in logstash are libvirtd.txt
    and libvirt, the index config looked for libvirtd as the current
    setup does not appear to match - we only have {%logmessage%} for message
    content now. Setting search tag to 'libvirt'.

    Change-Id: I27ae3b034581571285f42dab674ed85ff820cb6f
    Partial-bug: 1350121

[Patrick Crews (patrick-crews)]

NOTE: still seeing {%logmessage%} even with the change merged. Suspect something is wrong with the parsing pattern vs. the tags. Will dig into this a bit further.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/122466

[Patrick Crews (patrick-crews)]

The parsing pattern appeared to work perfectly for all the log lines I tested.
I did see grokparsefailure tags on other log files (apache logs), but the messages were still preserved.

[Patrick Crews (patrick-crews)]

libvirtd.txt and os-collect-config.txt appear to be good.
host-info.txt needs to be indexed - https://review.openstack.org/#/c/107841/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/122466
Committed: https://git.openstack.org/cgit/openstack-infra/config/commit/?id=6880db476172eceb17431e73bc1b06f672d321c8
Submitter: Jenkins
Branch: master

commit 6880db476172eceb17431e73bc1b06f672d321c8
Author: Derek Higgins <email address hidden>
Date: Thu Sep 18 17:30:47 2014 +0100

    Add syslog tag to os-collect-config logs

    We now have consistent os-collect-config log formats for both
    fedora and ubuntu tripleo ci jobs(from systemd journal and syslog
    respectively), see I27ea3d465670277ef1ddf3d1b3b9d52df4162807 and
    Ib8c1fc39d56b2b9c6d8e9b64a868def619aa2f1f. Tagging them as syslog
    will ensure they are handled by the logstash syslog grok filter,
    they currently don't match anything which results in messages
    being indexed with a value of "%{logmessage}".

    Change-Id: Iec5ffcb08e5b45fb01db25e14c943f000ca97a83
    Closes-Bug: #1350121