logstash results don't always match up with actual logs

[Joe Gordon]

the following logstash query (@message:"ConnectionFailed: Connection to neutron failed: Maximum attempts reached" AND @fields.filename:"console.html" AND @fields.build_change:"45424" AND @fields.build_patchset:"8")

produces a hit on 2013-09-13T10:24:46.000 but when looking at the original log file (http://logs.openstack.org/24/45424/8/check/gate-tempest-devstack-vm-neutron/5f4e9d1/console.html) the message string is not there.

[Clark Boylan (cboylan)]

Attached is the bad logstash event in csv as exported by Kibana.

[Clark Boylan (cboylan)]

The log file linked to above was for a different test run on the same change and patchset. The test run that produced this failure is http://logs.openstack.org/24/45424/8/check/gate-tempest-devstack-vm-neutron/0150828/console.html. The bug here is that logstash/elasticsearch are not keeping track of the build log path or the build UUID so making a mapping from logstash to log file on the log server is difficult.

To fix this we should carry the build UUID in our log events and/or just include the entire path to the log file in the logstash log events.

[Clark Boylan (cboylan)]

This was fixed by including build UUIDs in the logstash events. Now each log event is uniquely traceable back to its log file and job run.