apache2 SEGV with multiple SSL sites

[Alex Bligh]

Apache2 crashes with multiple SSL sites.

[Impact]

Apache may SEGV on initialisation (and thus refuse to start) when using multiple SSL sites in a moderately complex configuration. Though the crash is caused by OCSP stapling code, it is not necessary for OCSP to be enabled to cause the problem. As the problem is caused by a memory address changing between reads of the config file, in theory any configuration with one SSL site could refuse to run, though in practice a degree of complexity appears to be necessary to cause sufficient memory allocation to trigger the crash.

The bug is thus serious as any SSL apache configuration may not load.

[Testcase]

See comment #1

[Regression Potential]

The most likely regression potential is a failure of OCSP to work properly. OCSP is relatively new and little used code, and hence is less well tested than other areas. Though the work was done upstream and has been approved by OCSP-familiar apache authors, it is possible a change to the OCSP code will cause some OCSP functionality defect. However, the comparative lack of use of OCSP (compared to SSL) means the impact of any such failure should be limited.

Detailed description follows:

When starting apache2 with multiple SSL sites I get a SEGV like this:

(gdb) bt
#0 0x00007ffff05faaf3 in ?? () from /usr/lib/apache2/modules/mod_ssl.so
#1 0x00007ffff29647a6 in int_free_ex_data (class_index=<optimized out>, obj=0x555555af7460, ad=0x555555af7488) at ex_data.c:522
#2 0x00007ffff2a05061 in x509_cb (operation=operation@entry=3, pval=pval@entry=0x7fffffffc218, it=it@entry=0x7ffff2cc0780 <X509_it>,
    exarg=exarg@entry=0x0) at x_x509.c:113
#3 0x00007ffff2a08fea in asn1_item_combine_free (pval=pval@entry=0x7fffffffc218, it=it@entry=0x7ffff2cc0780 <X509_it>, combine=combine@entry=0)
    at tasn_fre.c:173
#4 0x00007ffff2a091c5 in ASN1_item_free (val=val@entry=0x555555af7460, it=it@entry=0x7ffff2cc0780 <X509_it>) at tasn_fre.c:71
#5 0x00007ffff2a0514c in X509_free (a=a@entry=0x555555af7460) at x_x509.c:141
#6 0x00007ffff05ee0b8 in ssl_pphrase_Handle (s=s@entry=0x7ffff7fc1de0, p=p@entry=0x7ffff7fbf028) at ssl_engine_pphrase.c:275
#7 0x00007ffff05e3658 in ssl_init_Module (p=0x7ffff7ff0028, plog=<optimized out>, ptemp=0x7ffff7fbf028, base_server=0x7ffff7fc1de0)
    at ssl_engine_init.c:194
#8 0x00005555555aa2a9 in ap_run_post_config (pconf=0x7ffff7ff0028, plog=0x7ffff7fbd028, ptemp=0x7ffff7fbf028, s=0x7ffff7fc1de0) at config.c:103
#9 0x000055555558ae07 in main (argc=6, argv=0x7fffffffe5a8) at main.c:765

This is 100% repeatable.

This looks very like:
  https://bugzilla.redhat.com/show_bug.cgi?id=1074406

save that I am not using Auth at all. However, ssl itself requires the socache logic, so perhaps it has the same root cause.

Disabling a couple of SSL sites normally resolves the problem.

What I expected to happen: apache2 to start without SEGV
What actually happened: apache2 did not start due to SEGV

root@nimtest:/root# lsb_release -rd
Description: Ubuntu 14.04.1 LTS
Release: 14.04

root@nimtest:/root# apt-cache policy apache2-bin
apache2-bin:
  Installed: 2.4.7-1ubuntu4.1
  Candidate: 2.4.7-1ubuntu4.1
  Version table:
 *** 2.4.7-1ubuntu4.1 0
        500 http://gb.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.7-1ubuntu4 0
        500 http://gb.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

root@nimtest:/root# dpkg --list | egrep '\b(apache2|libssl|openssl)'
ii apache2 2.4.7-1ubuntu4.1 amd64 Apache HTTP Server
ii apache2-bin 2.4.7-1ubuntu4.1 amd64 Apache HTTP Server (binary files and modules)
ii apache2-data 2.4.7-1ubuntu4.1 all Apache HTTP Server (common files)
ii apache2-dbg 2.4.7-1ubuntu4.1 amd64 Apache debugging symbols
ii apache2-utils 2.4.7-1ubuntu4.1 amd64 Apache HTTP Server (utility programs for web servers)
ii libgnutls-openssl27:amd64 2.12.23-12ubuntu2.1 amd64 GNU TLS library - OpenSSL wrapper
ii libssl1.0.0:amd64 1.0.1f-1ubuntu2.5 amd64 Secure Sockets Layer toolkit - shared libraries
ii libssl1.0.0-dbg:amd64 1.0.1f-1ubuntu2.5 amd64 Secure Sockets Layer toolkit - debug information
ii openssl 1.0.1f-1ubuntu2.5 amd64 Secure Sockets Layer toolkit - cryptographic utility
ii python-openssl 0.13-2ubuntu6 amd64 Python 2 wrapper around the OpenSSL library

Modules in use:

root@nimtest:/root# ls -1 /etc/apache2/mods-enabled/
access_compat.load
alias.conf
alias.load
auth_basic.load
authn_core.load
authn_file.load
authz_core.load
authz_groupfile.load
authz_host.load
authz_user.load
autoindex.conf
autoindex.load
cgi.load
dbd.load
deflate.conf
deflate.load
dir.conf
dir.load
env.load
filter.load
headers.load
ident2.load
lbmethod_byrequests.load
mime.conf
mime.load
mpm_prefork.conf
mpm_prefork.load
negotiation.conf
negotiation.load
php5.conf
php5.load
proxy.conf
proxy.load
proxy_balancer.conf
proxy_balancer.load
proxy_http.load
reqtimeout.conf
reqtimeout.load
rewrite.load
setenvif.conf
setenvif.load
slotmem_shm.load
socache_shmcb.load
ssl.conf
ssl.load
status.conf
status.load
substitute.load
websocket.load
websocket_draft76.load

Here's a startup log plus 'bt full'

root@nimtest:/root# APACHE_LOCK_DIR=/var/lock/apache2 APACHE_RUN_USER=www-data gdb --args /usr/sbin/apache2 -k start -X -e Debug
GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/apache2...Reading symbols from /usr/lib/debug//usr/sbin/apache2...done.
done.
(gdb) run
Starting program: /usr/sbin/apache2 -k start -X -e Debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Fri Sep 05 19:20:45.569382 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module access_compat_module from /usr/lib/apache2/modules/mod_access_compat.so
[Fri Sep 05 19:20:45.572221 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module alias_module from /usr/lib/apache2/modules/mod_alias.so
[Fri Sep 05 19:20:45.574731 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module auth_basic_module from /usr/lib/apache2/modules/mod_auth_basic.so
[Fri Sep 05 19:20:45.577289 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module authn_core_module from /usr/lib/apache2/modules/mod_authn_core.so
[Fri Sep 05 19:20:45.579829 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module authn_file_module from /usr/lib/apache2/modules/mod_authn_file.so
[Fri Sep 05 19:20:45.582459 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module authz_core_module from /usr/lib/apache2/modules/mod_authz_core.so
[Fri Sep 05 19:20:45.585176 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module authz_groupfile_module from /usr/lib/apache2/modules/mod_authz_groupfile.so
[Fri Sep 05 19:20:45.587851 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module authz_host_module from /usr/lib/apache2/modules/mod_authz_host.so
[Fri Sep 05 19:20:45.590460 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module authz_user_module from /usr/lib/apache2/modules/mod_authz_user.so
[Fri Sep 05 19:20:45.593611 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module autoindex_module from /usr/lib/apache2/modules/mod_autoindex.so
[Fri Sep 05 19:20:45.596597 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module cgi_module from /usr/lib/apache2/modules/mod_cgi.so
[Fri Sep 05 19:20:45.599459 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module dbd_module from /usr/lib/apache2/modules/mod_dbd.so
[Fri Sep 05 19:20:45.602970 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module deflate_module from /usr/lib/apache2/modules/mod_deflate.so
[Fri Sep 05 19:20:45.607589 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module dir_module from /usr/lib/apache2/modules/mod_dir.so
[Fri Sep 05 19:20:45.611506 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module env_module from /usr/lib/apache2/modules/mod_env.so
[Fri Sep 05 19:20:45.615353 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module filter_module from /usr/lib/apache2/modules/mod_filter.so
[Fri Sep 05 19:20:45.618714 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module headers_module from /usr/lib/apache2/modules/mod_headers.so
[Fri Sep 05 19:20:45.621296 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module ident_module from /usr/lib/apache2/modules/mod_ident2.so
[Fri Sep 05 19:20:45.624291 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module lbmethod_byrequests_module from /usr/lib/apache2/modules/mod_lbmethod_byrequests.so
[Fri Sep 05 19:20:45.627441 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module mime_module from /usr/lib/apache2/modules/mod_mime.so
[Fri Sep 05 19:20:45.630890 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module mpm_prefork_module from /usr/lib/apache2/modules/mod_mpm_prefork.so
[Fri Sep 05 19:20:45.634628 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module negotiation_module from /usr/lib/apache2/modules/mod_negotiation.so
[Fri Sep 05 19:20:45.771556 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module php5_module from /usr/lib/apache2/modules/libphp5.so
[Fri Sep 05 19:20:45.776446 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module proxy_module from /usr/lib/apache2/modules/mod_proxy.so
[Fri Sep 05 19:20:45.779712 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module proxy_balancer_module from /usr/lib/apache2/modules/mod_proxy_balancer.so
[Fri Sep 05 19:20:45.783012 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module proxy_http_module from /usr/lib/apache2/modules/mod_proxy_http.so
[Fri Sep 05 19:20:45.785965 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module reqtimeout_module from /usr/lib/apache2/modules/mod_reqtimeout.so
[Fri Sep 05 19:20:45.789713 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module rewrite_module from /usr/lib/apache2/modules/mod_rewrite.so
[Fri Sep 05 19:20:45.792660 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module setenvif_module from /usr/lib/apache2/modules/mod_setenvif.so
[Fri Sep 05 19:20:45.795575 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module slotmem_shm_module from /usr/lib/apache2/modules/mod_slotmem_shm.so
[Fri Sep 05 19:20:45.798641 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module socache_shmcb_module from /usr/lib/apache2/modules/mod_socache_shmcb.so
[Fri Sep 05 19:20:45.808987 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module ssl_module from /usr/lib/apache2/modules/mod_ssl.so
[Fri Sep 05 19:20:45.812723 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module status_module from /usr/lib/apache2/modules/mod_status.so
[Fri Sep 05 19:20:45.816188 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module substitute_module from /usr/lib/apache2/modules/mod_substitute.so
[Fri Sep 05 19:20:45.820615 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module websocket_module from /usr/lib/apache2/modules/mod_websocket.so
[Fri Sep 05 19:20:45.825187 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module websocket_draft76_module from /usr/lib/apache2/modules/mod_websocket_draft76.so
[Fri Sep 05 19:20:45.829577 2014] [so:debug] [pid 46103] mod_so.c(266): AH01575: loaded module websocket_vnc_proxy_module from /usr/lib/apache2/modules/mod_websocket_vnc_proxy.so
AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-extility-amber-listen.conf:10
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
[New Thread 0x7fffe6d32700 (LWP 46111)]
[Thread 0x7fffe6d32700 (LWP 46111) exited]

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff05faaf3 in ?? () from /usr/lib/apache2/modules/mod_ssl.so
(gdb) bt full
#0 0x00007ffff05faaf3 in ?? () from /usr/lib/apache2/modules/mod_ssl.so
No symbol table info available.
#1 0x00007ffff29647a6 in int_free_ex_data (class_index=<optimized out>, obj=0x555555af7460, ad=0x555555af7488) at ex_data.c:522
        mx = 1
        i = 0
        item = 0x555555835a90
        ptr = <optimized out>
        storage = 0x555555af6fa0
#2 0x00007ffff2a05061 in x509_cb (operation=operation@entry=3, pval=pval@entry=0x7fffffffc218, it=it@entry=0x7ffff2cc0780 <X509_it>,
    exarg=exarg@entry=0x0) at x_x509.c:113
        ret = 0x555555af7460
#3 0x00007ffff2a08fea in asn1_item_combine_free (pval=pval@entry=0x7fffffffc218, it=it@entry=0x7ffff2cc0780 <X509_it>, combine=combine@entry=0)
    at tasn_fre.c:173
        tt = <optimized out>
        seqtt = <optimized out>
        ef = <optimized out>
        cf = <optimized out>
        aux = <optimized out>
        asn1_cb = 0x7ffff2a04fa0 <x509_cb>
        i = <optimized out>
#4 0x00007ffff2a091c5 in ASN1_item_free (val=val@entry=0x555555af7460, it=it@entry=0x7ffff2cc0780 <X509_it>) at tasn_fre.c:71
No locals.
#5 0x00007ffff2a0514c in X509_free (a=a@entry=0x555555af7460) at x_x509.c:141
No locals.
#6 0x00007ffff05ee0b8 in ssl_pphrase_Handle (s=s@entry=0x7ffff7fc1de0, p=p@entry=0x7ffff7fbf028) at ssl_engine_pphrase.c:275
        using_cache = 0
        mc = 0x7ffff7fc5950
        sc = 0x7fffeb618bc8
        pServ = 0x7fffeb64c5b8
        cpVHostID = 0x7ffff7e5a160 "nimtest2.amb.flexiant.com:20443"
        szPath = "/etc/extility/skyline/ssl/certs/extility-skyline-cp-f1e48937-f534-39a7-80d6-7f9b7f790dc1.crt\000\177\000\000\060\313\377\377\377\177\000\000+", '\000' <repeats 11 times>, "\b", '\000' <repeats 51 times>, "\247\320\327\357\377\177", '\000' <repeats 42 times>...
        pPrivateKey = <optimized out>
        asn1 = <optimized out>
        ucp = 0x555555835785 ""
        length = <optimized out>
        pX509Cert = 0x555555af7460
        bReadable = <optimized out>
        aPassPhrase = 0x7ffff7e5a110
        nPassPhrase = 0
        nPassPhraseCur = -120945919
        cpPassPhraseCur = 0xc609ea311eb6e53b <error: Cannot access memory at address 0xc609ea311eb6e53b>
        nPassPhraseRetry = <optimized out>
        nPassPhraseDialog = 0
        nPassPhraseDialogCur = 272216735
        bPassPhraseDialogOnce = 1963170297
        cpp = <optimized out>
        i = 0
        j = 0
        algoCert = 1
        algoKey = 0
        at = <optimized out>
        an = 0x7ffff05fd3c2 "RSA"
        pkey_mtime = 0
        rv = <optimized out>
#7 0x00007ffff05e3658 in ssl_init_Module (p=0x7ffff7ff0028, plog=<optimized out>, ptemp=0x7ffff7fbf028, base_server=0x7ffff7fc1de0)
    at ssl_engine_init.c:194
        mc = <optimized out>
        sc = <optimized out>
        s = 0x0
#8 0x00005555555aa2a9 in ap_run_post_config (pconf=0x7ffff7ff0028, plog=0x7ffff7fbd028, ptemp=0x7ffff7fbf028, s=0x7ffff7fc1de0) at config.c:103
---Type <return> to continue, or q <return> to quit---
        pHook = 0x7ffff7eb2bb8
        n = 14
        rv = 0
#9 0x000055555558ae07 in main (argc=6, argv=0x7fffffffe5a8) at main.c:765
        c = 101 'e'
        showcompile = 0
        showdirectives = 0
        confname = 0x5555555ca607 "apache2.conf"
        def_server_root = 0x5555555ca5fa "/etc/apache2"
        temp_error_log = 0x0
        error = <optimized out>
        process = 0x7ffff7ff2118
        pconf = 0x7ffff7ff0028
        plog = 0x7ffff7fbd028
        ptemp = 0x7ffff7fbf028
        pcommands = 0x7ffff7fc7028
        opt = 0x7ffff7fc7118
        rv = <optimized out>
        mod = 0x5555557ec160 <ap_prelinked_modules+64>
        opt_arg = 0x7fffffffe831 "Debug"
        signal_server = <optimized out>

[Alex Bligh (ubuntu-alex-org)]

I think I've got about the minimal case for replication. Attached is a tiny perl script which generates a number of SSL sites of the form:

<VirtualHost 127.0.0.1:$port>
    ServerName 127.0.0.1:$port

    SSLEngine on
    SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
    SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

    DBDriver pgsql
</VirtualHost>

When the numebr of sites exceeds 61 (on my machine), I get an illegal instruction error.

The "DBDriver pgsql" itself is important, but I don't think this is a DBD problem. About anything that loads a module causes a problem.

[Alex Bligh (ubuntu-alex-org)]

Actually "DBDriver pgsql" causes the issue, but not "DBDriver mysql", and it can be outside the virtual host block. So I think this might be a pgsql driver issue.

Reported upstream at:
https://issues.apache.org/bugzilla/show_bug.cgi?id=56919

[Alex Bligh (ubuntu-alex-org)]

The number of sites required appears to vary. Also it appears to be necessary to have mod php5 enabled.

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better, and for filing the upstream bug and investigation further.

I'm worried about the regression risk of pushing for an update to 2.4.10 in Ubuntu; the changelog looks scary here, I can see some entries that suggest that there may be changes in behaviour and I don't want users yelling at me for causing regressions (in any case, only the Technical Board can approve this).

We can however push a minimal patch to Trusty and Utopic that fixes the issue, if you can identify one. A bisection might help you here. I see that you've provided the test case; I'd prefer for you to verify any fixes with those test cases if possible.

[Alex Bligh (ubuntu-alex-org)]

Robie: that attitude is quite understandable. I'm willing to do some work bisecting it, but I fear the root problem is going to be that addressed this commit:
http://svn.apache.org/viewvc?view=revision&revision=1573360
The ssl_pphrase_Handle routine is misleadingly named, and in fact is pretty much the core SSL initialisation routine for all the sites. What appears to be going wrong is one of the addresses for the callback going awry. The above commit rewrites this completely (which is an intrusive change) - the author's opinion of the previous code is evident from the commit message. As you can see, upstream's proposed fix was 'upgrade'. I don't think this will qualify as a 'minimal patch'.

As far as I can tell from playing so far, the root problem seems to be connected to .so file loading. modphp + moddbd postgresql tickles it, but I suspect other combinations will as well.

If it's going to be difficult to fix this against 2.4.7, would getting 2.4.10 (the Utopic version) into trusty-backports be permissible? That way at least I'd get security updates. I can confirm this builds out of the box with no issues.

[Alex Bligh (ubuntu-alex-org)]

Turns out 2.4.10 also has the bug after all (it's just more difficult to trigger). I think I have found the root cause. I've put details upstream.

[Robie Basak (racb)]

Thank you for the detailed investigation upstream.

> If it's going to be difficult to fix this against 2.4.7, would getting 2.4.10 (the Utopic version) into trusty-backports be permissible? That way at least I'd get security updates. I can confirm this builds out of the box with no issues.

I think trusty-backports would be admissible, but note that security updates to trusty-backports are done by community members (like universe is), and isn't managed by Canonical.

I guess we'll need to wait and see what the correct upstream fix is. But now that you've isolated it, it looks like a fix would be more isolated and suitable for cherry-picking?

[Alex Bligh (ubuntu-alex-org)]

Robie: removing the reference to certinfo_free where X509_get_ex_new_index is called within ssl_stapling_ex_init works around the 2.4.10 bug at the expense of a memory leak. I haven't (yet) verified this entirely fixes 2.4.7 though I suspect it will. I'll test that in a bit.

Obviously this solution is pretty foul, but is probably better than the current situation. A better solution from upstream would be welcomed.

The underlying issue is that not all SSL resources are being correctly individually freed, and for various reasons the cleanup function can't be used to clean them all up. If I've understood this bug right, any apache config that uses SSL is vulnerable to a crash on startup; it just needs to be reasonably complex (sufficiently complex to cause dlopen() to choose a different memory address to load the SSL module).

[Alex Bligh (ubuntu-alex-org)]

I can confirm that the above workaround fixes 2.4.7, both my testcase and our real world version. I attach a patch. This is probably 'better than nothing'.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Patch to avoid calling certinfo_free (ugly workaround)" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Robie Basak (racb)]

Thanks Alex. I'd prefer to wait to see if a proper fix is committed upstream in the next few weeks, so as we don't have to risk regressions to Trusty users twice (and go through the SRU process twice, etc).

If a fix doesn't happen "soon", then maybe we should push your workaround back to Trusty, as I agree that pragmatically a leak is better than a crash, particularly if the leak only happens on startup.

[Alex Bligh (ubuntu-alex-org)]

The fix for this is now committed in trunk. A 2.4 backport is available. See:

https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/STATUS?r1=1631030&r2=1631029

Patch (per the above) at:

https://people.apache.org/~kbrand/mod_ssl-2.4.x-PR54357.diff

[Alex Bligh (ubuntu-alex-org)]

This has now been merged into 2.4. See https://issues.apache.org/bugzilla/show_bug.cgi?id=54357

Any chance this can now be backported to Trusty? The impact is pretty severe.

[Robie Basak (racb)]

> Any chance this can now be backported to Trusty? The impact is pretty severe.

It sounds like a good candidate, though I haven't reviewed the patch yet. I'm away at the moment, so if somebody else wants to work on this in the meantime, please feel free. The process is documented at https://wiki.ubuntu.com/StableReleaseUpdates#Procedure.

Otherwise please do poke me in a couple of weeks if I haven't reported any progress.

[Alex Bligh (ubuntu-alex-org)]

I have attached a backport to 2.4.7 to this comment. This is a backport of the backport to 2.4.x in upstream svn. More details in the commit message.

This is a straight patch to the source (produced from git) rather than a proper packaged up patch, if you see what I mean.

I've put this up on github too for ease of review:
   https://github.com/abligh/apache2-2.4.7-ubuntu-trusty/commit/6e24e496c7aee8aa1ff13a41dae71c91fe8c0bbe

[Alex Bligh (ubuntu-alex-org)]

I have added [Impact] and [Regression potential] sections.

Do the SRU requirements mean we need a patch for U too? I'm not sure what "current development release" means right now given that U is out. I believe the upstream 2.4.10 patch should apply straight to U. It's upstream, so V will presumably get whatever upstream has.

I'll ask for someone to nominate this but I think you may need to take it from here, Robie.

[Alex Bligh (ubuntu-alex-org)]

Robie: this is me poking you after a couple of weeks, as requested.

[Robie Basak (racb)]

Thanks Alex. I'm sorry I've been slow. I'm still not back at work as normal but I'll try to look at this now.

Just to log what I've seen so far:

Looks like Vivid will need to either cherry-pick this, or a merge may be sufficient since your message says you picked r1629372, r1629485, r1629519 and Debian 2.4.10-6 reports to have picked everything up to r1632831 but I need to check this.

So I think what I'll probably do first is merge Vivid to meet SRU requirements for Trusty, and then take your patch, adapt it to quilt and apply it to Trusty.

To prevent a future regression, would be you able to check Vivid once I have merged and uploaded it please, to ensure the bug is fixed there?

[Alex Bligh (ubuntu-alex-org)]

Robie: no apology needed, and yes I would be happy to check Vivid.

[Stefan Fritsch (sf-sfritsch)]

> Looks like Vivid will need to either cherry-pick this, or a merge may be sufficient
> since your message says you picked r1629372, r1629485, r1629519 and Debian
> 2.4.10-6 reports to have picked everything up to r1632831 but I need to check this.

The commits mentioned by Alex are in the trunk branch, but 2.4.10-6 merged only from the 2.4.x branch. The relevant fix in the 2.4.x branch is r1634529. I will include that commit in 2.4.10-8.

[Robie Basak (racb)]

Thanks Stefan, I didn't consider that.

I started with a merge of 2.4.10-7 that's now stuck in vivid-proposed due to bug 1393832 which I've just filed. I could re-merge 2.4.10-8 though, and then continue with the SRU - no need to block the SRU on this.

[Robie Basak (racb)]

2.4.10-8ubuntu1 is now in vivid-proposed and should fix this bug for Vivid and for future releases, but it won't land in Vivid itself until bug 1393832 is fixed. I'd like to focus on this SRU before working on that bug.

Alex, could you please verify that the bug is fixed in vivid-proposed for you? Plain Vivid won't work. I suggest that you update to latest vivid, enable vivid-proposed and upgrade apache2's binary packages only to test, to avoid pulling various unrelated things in.

So next, the SRU for this. This is a pretty extensive thing to SRU, but given the severity of the bug I think we should do it. I've spoken to Chris Arges (SRU team) and we considered options, and settled on using your backport provided that I give it a careful review, so I need to do this next. Chris will still have the final call though after I've uploaded.

I'm not sure I'll manage to get it done today though (still at a sprint) and I'm on vacation next week. If I don't report back today, if another Ubuntu dev with upload rights wants to take on the review next week, please go ahead without any need to coordinate with me.

Alex, thanks for your patience on this. I appreciate the work you've already put into this. One of the things we discussed at this sprint was on onboarding more developers to be able to take this sort of work on, so I hope that it won't take so long to process next time.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.10-8ubuntu2

---------------
apache2 (2.4.10-8ubuntu2) vivid; urgency=medium

  * Allow "triggers-awaited" and "triggers-pending" states in addition to
    "installed" when determining whether to defer actions or process
    deferred actions (LP: #1393832).
 -- Colin Watson <email address hidden> Wed, 26 Nov 2014 11:31:44 +0000

[Alex Bligh (ubuntu-alex-org)]

Robie: I've verified that the Vivid version works fine. Can I ping you re getting the SRU done for Trusty?

[Alex Bligh (ubuntu-alex-org)]

Robie: can I ping you once more re the backport to trusty?

[Alex Bligh (ubuntu-alex-org)]

Any update on this one?

[Robie Basak (racb)]

I've spent a few hours over the last couple of days reviewing this in detail. I've gone over Alex's proposed patch to Trusty carefully, making sure I understand every line in the context of the existing code. I've also carefully gone through upstream's review comments, and upstream's commit to their 2.4 branch (based on a more recent version which necessitated Alex's backport). I've compared upstream's patch to Alex's patch and verified his backport.

I haven't found any issues after quite some time looking for them.

Although the patch is more invasive than I'd like for an SRU, the refactoring he did is the most reasonable way to fix the memory corruption issues in this bug. The only other option to get a more minimal patch would be to leave a memory leak on reload.

This doesn't completely eliminate the regression risk with this kind of update, but I have reviewed as carefully as I can to mitigate it.

+1 for Alex's patch. There are some minor whitespace, comment, constant and name changes that aren't strictly minimal, but I think that given the complexity of the patch it's safer to leave them as-is as they've come directly from the upstream backport. The only code touched is in the areas that needed touching anyway.

I have uploaded this to trusty-proposed and this now awaits review from the SRU team. As Chris Arges looked at this previously, I'll ask him if he can look at this again now.

I recommend that for SRU verification, in addition to checking for crashes, we test SSL functionality carefully - both with and without OCSP certificates if possible.

Top quality work, Alex, especially considering the complexity involved. Thank you.

[Alex Bligh (ubuntu-alex-org)]

Thanks Robie.

If it helps, we have been running this patch on many tens of machines of machines since early Nov 2014 (so approximately 4 months) without any ill effects, with and without SSL (though we don't use stapling).

[Brian Murray (brian-murray)]

Hello Alex, or anyone else affected,

Accepted apache2 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.7-1ubuntu4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Alex Bligh (ubuntu-alex-org)]

Thanks. Verified that this works with the original test cases, and marked verification-done.

[Alex Bligh (ubuntu-alex-org)]

http://people.canonical.com/~ubuntu-archive/pending-sru.html indicates there is allegedly a regression in svn. Last build is here: https://jenkins.qa.ubuntu.com/job/trusty-adt-subversion/lastBuild/ARCH=amd64,label=adt/ and indeed the build log shows a failure here: https://jenkins.qa.ubuntu.com/job/trusty-adt-subversion/lastBuild/ARCH=amd64,label=adt/artifact/results/log

cjwatson suggested on #ubuntu-devel:

"This might just mean that the fix for https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1393832 needs to be cherry-picked as well, but I'm not sure. Perhaps rbasak can investigate."

I tried replicating this with adt locally, but can't get it to fail the test either before OR after the change. I'd suggest that the test suite running would have failed both before and after this change. If this is indeed https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1393832 it will need someone to propose an SRU for it. My feeling is however that it is unrelated to this change.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.7-1ubuntu4.4

---------------
apache2 (2.4.7-1ubuntu4.4) trusty-security; urgency=medium

  * SECURITY UPDATE: HTTP header replacement via HTTP trailers (LP: #1425141)
    - debian/patches/CVE-2013-5704.patch: don't merge trailers by default
      and add a "MergeTrailers" directive to revert to previous behaviour
      to include/http_core.h, include/httpd.h, modules/http/http_filters.c,
      modules/http/http_request.c, modules/loggers/mod_log_config.c,
      modules/proxy/mod_proxy_http.c, server/core.c, server/protocol.c.
    - CVE-2013-5704
  * SECURITY UPDATE: mod_cache denial of service via empty HTTP
    Content-Type header
    - debian/patches/CVE-2014-3581.patch: check for NULL in
      modules/cache/cache_util.c.
    - CVE-2014-3581
 -- Marc Deslauriers <email address hidden> Tue, 10 Mar 2015 07:42:50 -0400

[Alex Bligh (ubuntu-alex-org)]

Thanks for everyone's work on this - much appreciated.