some files are not getting added to logstash.openstack.org

[Joe Gordon]

http://logstash.openstack.org/#eyJzZWFyY2giOiJidWlsZF9zaG9ydF91dWlkOlwiOTdmMmNkZVwiIEFORCAgdGFnczpzY3JlZW4tbioiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDg0MDc0NTY2MjUsIm1vZGUiOiJ0cmVuZCIsImFuYWx5emVfZmllbGQiOiJmaWxlbmFtZSJ9

Some jobs have missing logs such as the link above.

[Joe Gordon (jogo)]

http://logs.openstack.org/77/109177/2/gate/gate-tempest-dsvm-full/97f2cde/logs/

[Clark Boylan (cboylan)]

I think part of the issue here is the * isn't working as expected in tags? build_short_uuid:"97f2cde" AND NOT filename:"console.html" AND NOT filename:"logs/syslog.txt" AND NOT filename:"logs/screen-n-cond.txt" AND NOT filename:"logs/screen-g-reg.txt" AND NOT filename:"logs/screen-h-api-cw.txt" is a query that returns 0 results and the files NOTed out there do return data.

[Clark Boylan (cboylan)]

From logstash-worker04 C worker:

2014-08-18 16:46:55,412 Unable to get log data.

I think logs.o.o's apache process was unable to serve those files at that time for some reason. We should see if we can catch this in action.

[Clark Boylan (cboylan)]

After looking on the logs.o.o apache side there is nothing logged in the access log for these requests. This is potentially network disruption between logs.o.o and our logstack worker nodes.

Please update this bug if we can find more incidences of it.