Reduce zlib compression level for massive performance increase

[Anders Kaseorg]

When apport takes a core dump of a large application, it uses 100% CPU for a _long_ time (often 30 seconds or more) to compress the core dump. The system is virtually unusable during this time, especially if the application was the window manager and won’t be restarted until the core dump is complete.

This is a longstanding known problem; it motivated the switch from bzip2 to zlib level 9 back in apport 0.24. That made it much better, but it’s still quite bad. So why not switch from zlib level 9 to zlib level 1?

I tested this on a core dump of gnome-shell, and while it increased the size of the .crash file by 24% (68.5 MB to 85.0 MB), it decreased the needed CPU time by a FACTOR OF EIGHT (58.6 seconds to 7.3 seconds). This tradeoff seems more than worthwhile for this use case.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin Pitt (pitti)]

How does it do with the default level of 5? 24% size increase is a lot, and uploading 30 MB extra is not trivial. So "5" might provide a better compromise here?

[Anders Kaseorg (andersk)]

Level 5 isn’t crazy, though it’s still some 70% slower than level 1.

zlib.compressobj(level=1) time=7.6 size=84869397
zlib.compressobj(level=2) time=7.8 size=82465461
zlib.compressobj(level=3) time=8.4 size=79576183
zlib.compressobj(level=4) time=12.2 size=74672475
zlib.compressobj(level=5) time=13.2 size=72642875
zlib.compressobj(level=6) time=15.7 size=69988031
zlib.compressobj(level=7) time=17.9 size=69281853
zlib.compressobj(level=8) time=30.4 size=68666299
zlib.compressobj(level=9) time=59.0 size=68326007

bz2.BZ2Compressor(compresslevel=1) time=30.6 size=61319073
bz2.BZ2Compressor(compresslevel=9) time=40.0 size=54246781

lz4.compress time=2.4 size=108683033

Sizes are after base64 encoding.

(If you’re worried about upload size, why not recompress the report at upload time, when it’s not going to take the user by surprise?)

[Anders Kaseorg (andersk)]

Ping? It’s still really frustrating to have the entire system go unresponsive for a minute just because gnome-shell crashed and apport won’t let it restart until it’s finished compressing the core dump.

If you’re not willing to go all the way to zlib level 1, can we at least proceed with going down to 5 like you suggested?

Alternatively, python3-brotli entered Ubuntu shortly after my original report. brotli level 2 compresses faster than zlib level 1 with smaller output than zlib level 9, while brotli level 1 is as almost as fast as lz4 with output size like zlib level 5!

I don’t have the crash dump I used for the tests above, so here are some tests on a somewhat smaller one.

zlib.compressobj(level=1) time=7.1 size=41822029
zlib.compressobj(level=2) time=7.1 size=40718379
zlib.compressobj(level=3) time=7.4 size=39638807
zlib.compressobj(level=4) time=13.1 size=31197301
zlib.compressobj(level=5) time=13.9 size=30423373
zlib.compressobj(level=6) time=15.1 size=29763741
zlib.compressobj(level=7) time=16.3 size=29526183
zlib.compressobj(level=8) time=25.6 size=29191955
zlib.compressobj(level=9) time=39.2 size=29067237

bz2.BZ2Compressor(compresslevel=1) time=25.0 size=25198553
bz2.BZ2Compressor(compresslevel=9) time=30.6 size=23280287

lz4.compress time=0.9 size=50649857
lz4.compressHC time=5.4 size=40462669

brotli.Compressor(quality=1) time=1.2 size=30559841
brotli.Compressor(quality=2) time=3.8 size=26162317
brotli.Compressor(quality=3) time=6.2 size=26256343
brotli.Compressor(quality=4) time=8.7 size=24982273
brotli.Compressor(quality=5) time=29.2 size=21744749
brotli.Compressor(quality=6) time=45.5 size=21502521
brotli.Compressor(quality=7) time=81.7 size=21355153
brotli.Compressor(quality=8) time=145.9 size=21249997

If a crash could be recorded in a second instead of a minute with almost no size difference, I’d call that a win.

[Brian Murray (brian-murray)]

https://bazaar.launchpad.net/~apport-hackers/apport/trunk/revision/3240

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.20.11-0ubuntu2

---------------
apport (2.20.11-0ubuntu2) eoan; urgency=medium

  * test/test_backend_apt_dpkg.py: as of r3211 an error is not raised if the
    binary is not found. Cherry picked from upstream r3248.
  * backends/packaging-apt-dpkg.py: writing the contents mapping pickle file
    can Traceback on systems with little memory so don't write it. Cherry
    picked from upstream r3250.
  * test/test_apport_valgrind.py: skip test_sandbox_cache_options if system
    has little memory.

 -- Brian Murray <email address hidden> Mon, 20 May 2019 18:39:46 -0700