The logstash gearman workers should properly handle gzipped files.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Core Infrastructure |
Fix Released
|
Low
|
Clark Boylan |
Bug Description
Today the logstash gearman workers have hacked around dealing with gzipped content. They rely on the file extension of the requested file to know when the file is gzipped. Instead they should rely on Content-Type and Content-Encoding http headers to know when the content is gzipped.
Fixing this will probably happen in 2 distinct chunks. First handle Content-Type being gzip so that the new devstack log annotation stuff doesn't break logstash. Second handle Content-Encoding being gzip so that the worker can tell the web server it accepts gzipped type and get all of its content responses gzipped. This second bit is a little tricky because the console.html files may not have finished copying by the time we have requested them. To work around this we keep requesting bytes until an end of file pattern is recognized. Gzip content can't be streamed easily so the worker code will need to be updated to properly ungzip the file and check for EOF.
Changed in openstack-ci: | |
assignee: | nobody → Clark Boylan (cboylan) |
status: | Triaged → In Progress |
Reviewed: https:/ /review. openstack. org/39547 github. com/openstack- infra/config/ commit/ a20c139f3d9b423 9943d806ebbfdca d337509961
Committed: http://
Submitter: Jenkins
Branch: master
commit a20c139f3d9b423 9943d806ebbfdca d337509961
Author: Clark Boylan <email address hidden>
Date: Wed Jul 31 11:23:19 2013 -0700
Handle html log annotations.
* modules/ openstack_ project/ files/logstash/ log-gearman- worker. py: worker. py needs to check the
The annotated logs served by logs-dev and soon to be served to logs.o.o
return .txt files gzipped. log-gearman-
Content-Type in the reponse headers to see if the txt files were gzipped
in order to properly handle this.
partial-bug: #1207047 a6e3d20e8369e40 7df151143ff
Change-Id: I5981cde145a572