The logstash gearman workers should properly handle gzipped files.

[Clark Boylan]

Today the logstash gearman workers have hacked around dealing with gzipped content. They rely on the file extension of the requested file to know when the file is gzipped. Instead they should rely on Content-Type and Content-Encoding http headers to know when the content is gzipped.

Fixing this will probably happen in 2 distinct chunks. First handle Content-Type being gzip so that the new devstack log annotation stuff doesn't break logstash. Second handle Content-Encoding being gzip so that the worker can tell the web server it accepts gzipped type and get all of its content responses gzipped. This second bit is a little tricky because the console.html files may not have finished copying by the time we have requested them. To work around this we keep requesting bytes until an end of file pattern is recognized. Gzip content can't be streamed easily so the worker code will need to be updated to properly ungzip the file and check for EOF.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/39547
Committed: http://github.com/openstack-infra/config/commit/a20c139f3d9b4239943d806ebbfdcad337509961
Submitter: Jenkins
Branch: master

commit a20c139f3d9b4239943d806ebbfdcad337509961
Author: Clark Boylan <email address hidden>
Date: Wed Jul 31 11:23:19 2013 -0700

    Handle html log annotations.

    * modules/openstack_project/files/logstash/log-gearman-worker.py:
    The annotated logs served by logs-dev and soon to be served to logs.o.o
    return .txt files gzipped. log-gearman-worker.py needs to check the
    Content-Type in the reponse headers to see if the txt files were gzipped
    in order to properly handle this.

    partial-bug: #1207047
    Change-Id: I5981cde145a572a6e3d20e8369e407df151143ff

[Clark Boylan (cboylan)]

I think they now handle this as best they can. They request gzipped content first, if available they use the gzipped content, if not available the fall back on uncompressed content. The wsgi log modifier app makes doing this all in one request somewhat problematic, but for the majority of cases we should get what we want on the first request.