Add support for compressed cores

[Haw Loeung]

Hi,

Currently, we have a pretty large backlog of cores to process, the cores directory is at 184G. Would it be possible to add support for compressing the cores?

Randomly picking out some of the larger cores and compressing (gzip with default compression level) them seems to suggest we'd save around 50%:

         compressed uncompressed ratio uncompressed_name
          732576993 3083538432 76.2% 1c94ac3e-47e8-11e2-9d14-e4115b0f8a4a.core
           15570174 220979200 93.0% 239a4308-4811-11e2-ab8e-2c768aafd08c.core
          827197319 1093217311 24.3% 2627d290-4878-11e2-8d54-2c768aafd08c
          837838177 1108439573 24.4% 41413548-4856-11e2-b5b4-2c768aafd08c
          960665967 1282747535 25.1% 4a83d5d8-482a-11e2-bb82-e4115b0f8a4a
          240154514 317718528 24.4% 52f12f5e-48a7-11e2-9eaf-e4115b0f8a4a
          680541715 2829602816 75.9% 8da28ff6-482c-11e2-9475-e4115b0f8a4a.core
          666078076 887166520 24.9% 8da28ff6-482c-11e2-9475-e4115b0f8a4a
          271885158 365674049 25.6% 932a1678-4882-11e2-b9ed-2c768aafd08c
          179878866 245787972 26.8% b44d72d2-4837-11e2-a3d5-e4115b0f8a4a
          129184094 171881716 24.8% becba9b4-489f-11e2-bd82-2c768aafd08c
         5541571053 11606753652 52.3% (totals)

I'm not sure how cores are uploaded, but if we could have the clients compress them, we'd also save on network traffic and more importantly, disk I/O.

[Evan (ev)]

Hi Haw,

The data we receive from the clients is gzip'ed and base64'ed (because apport wants to produce text-only report files).

This seems wasteful, as your findings suggest. We should be decoding the base64 data client-side and just sending the gzip data, or possibly using a different compression algorithm in apport (and whoopsie). I briefly looked at lz4, and its higher compression ratio option looks promising.

The *.core files are the uncompressed core files and are either leftover from crashed retracers, or retraces-in-progress.