"Mutex file:${APACHE_LOCK_DIR} default" should be disabled by default on Linux because it leads to errors

[Miranda Schumacher]

[Impact]
The default apache2.conf causes apache to issue streams of error
messages about deadlocks acquiring the SSL session cache lock.

Users are still reporting seeing this flaw in production (Xenial-based)
hosts.

[Test Case]
Reproduction steps TBD. Problem exhibits on high load systems. Verification will need to be done by those seeing the issue in production.

[Regression Potential]
Since this only changes the config installed by default, it won't impact existing installations, however behaviors to watch for would be SSL-related or configuration-related oddnesses.

[Fix]
Backport a fix applied in bionic and newer, that modifies the makes
Apache use pthread mutexes by default on Linux, or fctnl on other
architectures that lack robust pthread muxexes.

[Other Info]
Users should be aware that if they haven't changed /etc/apache2/apache2.conf this will automatically apply the fix, but users that have customized apache2.conf may still need to add it manually.

[Original Report]
OS:
Ubuntu 14.04 LTS

Kernel:
3.13.0-79-generic x86_64

Apache:
2.4.7-1ubuntu4.5

In the default Apache 2.4 config on Ubuntu 14.04 LTS is the following set in /etc/apache2/apache2.conf:

Mutex file:${APACHE_LOCK_DIR} default

(/debian/config-dir/apache2.conf in http://archive.ubuntu.com/ubuntu/pool/main/a/apache2/apache2_2.4.7-1ubuntu4.5.debian.tar.gz)

which leads to the following output of "apache2ctl -t -D DUMP_RUN_CFG":

Mutex default: dir="/var/lock/apache2" mechanism=fcntl

This leads constantly to a lot of these warning/emergency messages on a server with 200 busy worker threads, 100 Requests/s, 300 KB/s:

[Tue Mar 08 16:08:18.596653 2016] [ssl:warn] [pid 8339:tid 140182179256064] (35)Resource deadlock avoided: AH02026: Failed to acquire SSL session cache lock

[Wed Mar 09 07:09:31.099331 2016] [mpm_worker:emerg] [pid 26526:tid 139668485949184] (35)Resource deadlock avoided: AH00273: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.

Solution (as suggested by Yann Ylavic from Apache):
Commenting (removing) the Mutex directive, which leads to the following output of "apache2ctl -t -D DUMP_RUN_CFG":

Mutex default: dir="/var/run/apache2/" mechanism=default

Then, there are no error messages anymore.

For the discussion, see the corresponding Apache httpd-users mailing list thread:

http://httpd.markmail.org/message/c7w5aujfmy2kfazi

(thread subject 'Lots of messages "[ssl:warn] Resource deadlock avoided: AH02026: Failed to acquire SSL session cache lock"' from 2016-03-08)

Here some more information:

# apache2ctl -V
Server version: Apache/2.4.7 (Ubuntu)
Server built: Jul 24 2015 17:25:11
Server's Module Magic Number: 20120211:27
Server loaded: APR 1.5.1-dev, APR-UTIL 1.5.3
Compiled using: APR 1.5.1-dev, APR-UTIL 1.5.3
Architecture: 64-bit
Server MPM: worker
  threaded: yes (fixed thread count)
    forked: yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/etc/apache2"
 -D SUEXEC_BIN="/usr/lib/apache2/suexec"
 -D DEFAULT_PIDLOG="/var/run/apache2.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="mime.types"
 -D SERVER_CONFIG_FILE="apache2.conf"

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.23-7ubuntu1

---------------
apache2 (2.4.23-7ubuntu1) zesty; urgency=medium

  * Merge from Debian unstable. Remaining changes:
    - debian/{control, apache2.install, apache2-utils.ufw.profile,
      apache2.dirs}: Add ufw profiles.
    - debian/apache2.py, debian/apache2-bin.install: Add apport hook.
    - debian/rules: Fix cross-building by passing
      DEB_{HOST,BUILD}_GNU_TYPE to configure.
    - debian/patches/086_svn_cross_compiles: Backport several cross
      fixes from upstream
    - d/index.html, d/icons/ubuntu-logo.png, d/apache2.postrm: replace
      Debian with Ubuntu on default page.
    - Don't build experimental http2 module for LTS:
      + debian/control: removed libnghttp2-dev Build-Depends (in universe).
      + debian/config-dir/mods-available/http2.load: removed.
      + debian/rules: removed proxy_http2 from configure.
    - Correct systemd-sysv-generator behavior by customizing some
      parameters:
      + d/apache2-systemd.conf: add a drop-in file to specify some
        parameters for the systemd unit (type=Forking and
        RemainsAfterExit=no), this allow a correct state synchronisation
        between systemctl status and actual state of apache2 daemon.
      + d/apache2.install: place the apache2-systemd.conf file in the
        correct location.

apache2 (2.4.23-7) unstable; urgency=medium

  * Make apache2-dev depend on openssl 1.0, too. Closes: #844160
  * Move DefaultRuntimeDir and pid file for multi-instances to
    /var/run/apache2-xxx. Thanks to Horst Platz for the debugging.
    Closes: #838932 LP: #1627339
  * Fix systemd unit naming for multi-instances.
  * Tweak embedded .tar.gz some more to build reproducibly.

apache2 (2.4.23-6) unstable; urgency=medium

  * One more tweak for reproducible build. Thanks to Daniel Shahaf for the
    patch. Closes: #839977
  * Avoid building with openssl 1.1 for now. See #828236

apache2 (2.4.23-5) unstable; urgency=low

  * Team upload.

  [ Stefan Fritsch ]
  * Tweak creation of .tar.gz embedded in preinst to get reproducible
    build.

  [ Raphaël Hertzog ]
  * Add systemd unit files. Closes: #798430
  * Improve a2enmod to enable apache-htcacheclean with systemctl and let
    it enable '<email address hidden>' for multi-instance
    support.
  * Improve setup-instance to rely on the systemd <email address hidden> for
    multi-instance support.
  * Drop /lib/systemd/system/apache2.service.d/forking.conf now that we have
    proper native systemd support.
  * Modify handling of /etc/init.d/apache-htcacheclean to have a usual
    Default-Start value but instead we disable it manually in the postinst.
    That way "systemctl enable apache-htcacheclean" works.
  * Add some lintian overrides for non-problems (two update-rc.d calls in
    postinst, and a .js file with a very long line).

apache2 (2.4.23-4) unstable; urgency=medium

  * Fix pre-inst script for new installations. Closes: #834169

apache2 (2.4.23-3) unstable; urgency=low

  * Fix conffiles that may have got the wrong content during upgrade from
    wheezy to early jessie versions. Closes: #794933
  * Also restore re-introduced *.load fil...

[Haw Loeung (hloeung)]

Can we get the following backported to Xenial (so 2.4.18-2ubuntu3.14)?

| apache2 (2.4.20-1) unstable; urgency=medium
|
| * On Linux, use pthread mutexes. On kfreebsd/hurd, continue using fctnl
| because they lack robust pthred mutexes. LP: #1565744, #1527044

Seeing it on a couple of Xenial hosts and the fix being to comment out the following line from /etc/apache2/apache2.conf:

| Mutex file:${APACHE_LOCK_DIR} default

[Bryce Harrington (bryce)]

Haw Loeung, is there a simple way to reproduce the message?

[Bryce Harrington (bryce)]

I've packaged the fix for xenial in this branch:

    https://code.launchpad.net/~bryce/ubuntu/+source/apache2/+git/apache2/+ref/ubuntu/xenial-devel/

However I've not been able to construct a configuration+workload to reproduce the error message. Can someone describe a way to trigger the messages synthetically (e.g. in an lxc container)?

Meanwhile, setting this to incomplete and dropping the server-next tag.

[Haw Loeung (hloeung)]

Unfortunately, I'm not sure how best to reproduce this. I can tell you that we were seeing this on really busy units.

[Bryce Harrington (bryce)]

PPA is here:

  https://launchpad.net/~bryce/+archive/ubuntu/apache2-sru-lp1565744-apache-lock-dir

[Haw Loeung (hloeung)]

Installed packages from Bryce's PPA and confirmed no more '(35)Resource deadlock avoided: AH02026: Failed to acquire SSL session cache lock' logged.

See https://pastebin.ubuntu.com/p/bFJ9C5hFCD/

[Robie Basak (racb)]

For an SRU I think I'd have preferred a straight modification of apache2.conf in the source instead of backporting the conditional Linux vs !Linux behaviour from Debian, given that Ubuntu only runs on Linux. This would make the source patch easier to review. But since this should affect the source only, it's probably not worth changing that now. Instead, however, please could you run a debdiff against the old and new built binary debs during SRU verification to ensure that the change landing is indeed what we intend, as well as the normal SRU verification steps?

[Robie Basak (racb)]

Hello Miranda, or anyone else affected,

Accepted apache2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.18-2ubuntu3.16 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Haw Loeung (hloeung)]

Looks good to me, thanks:

| https://paste.ubuntu.com/p/D3BfjqzR4t/

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.18-2ubuntu3.17

---------------
apache2 (2.4.18-2ubuntu3.17) xenial-security; urgency=medium

  * SECURITY UPDATE: mod_rewrite redirect issue
    - debian/patches/CVE-2020-1927-1.patch: factor out default regex flags
      in include/ap_regex.h, server/core.c, server/util_pcre.c.
    - debian/patches/CVE-2020-1927-2.patch: add AP_REG_NO_DEFAULT to allow
      opt-out of pcre defaults in include/ap_regex.h,
      modules/filters/mod_substitute.c, server/util_pcre.c,
      server/util_regex.c.
    - CVE-2020-1927
  * SECURITY UPDATE: mod_proxy_ftp uninitialized memory issue
    - debian/patches/CVE-2020-1934.patch: trap bad FTP responses in
      modules/proxy/mod_proxy_ftp.c.
    - CVE-2020-1934

 -- Marc Deslauriers <email address hidden> Wed, 12 Aug 2020 17:35:50 -0400