openssl 1.1.1 memory overuse/leak

[Sergey]

[Impact]

At some point in the past do_ssl3_write() used to return the number of
bytes written, or a value <= 0 on error.

With libssl1.11 it now just returns a success/
error code and writes the number of bytes written to |tmpwrit|.

The SSL_MODE_RELEASE_BUFFERS code was still looking at the return code
for the number of bytes written rather than |tmpwrit|. This has the effect
that the buffers are not released when they are supposed to be.

Thus such software as nginx currenty use significantly more memory compared to libssl1.0.

[Test Case]

Use 'top' to measure the memory usage by nginx with ssl configured.

Example:

No memory overuse:
ii libssl1.1:amd64 1.1.0g-2ubuntu4 amd64 Secure Sockets Layer toolkit - shared libraries
https://launchpadlibrarian.net/442818951/1.1.0g.png

2,5x memory overuse:
ii libssl1.1:amd64 1.1.1-1ubuntu2.1~18.04.4 amd64 Secure Sockets Layer toolkit - shared libraries
https://launchpadlibrarian.net/442819146/1.1.1.png

[Regression Potential]

Low. This particular fix is tiny (https://git.openssl.org/?p=openssl.git;a=commitdiff;h=f2bb79a) and has been released and used in a couple of upstream openssl versions already without issue.

[Other Info]

The fix has been tested by nginx team and it solved the memory overuse issue.

[Alex Murray (alexmurray)]

Thanks for reporting this issue - this sounds like it might be suitable as a StableReleaseUpdate - in particular the Regressions section https://wiki.ubuntu.com/StableReleaseUpdates#Regressions sounds relevant in this case. Could you please adapt this bug report following the template in https://wiki.ubuntu.com/StableReleaseUpdates#Procedure?

[Adrien Nader (adrien)]

I expect the fix for this to be shipped already and Ubuntu 18.04 is reaching end-of-life in a few days. Considering both of these elements, I'm going to mark this as Fix Released.