Activity log for bug #1565744

Date Who What changed Old value New value Message
2016-04-04 11:26:19 Miranda Schumacher bug added bug
2016-06-15 16:12:04 Launchpad Janitor apache2 (Ubuntu): status New Confirmed
2016-11-16 17:03:33 Launchpad Janitor apache2 (Ubuntu): status Confirmed Fix Released
2016-11-16 17:03:33 Launchpad Janitor cve linked 2016-4979
2016-11-16 17:03:33 Launchpad Janitor cve linked 2016-5387
2017-01-30 13:36:36 Raphaël Droz bug added subscriber Raphaël Droz
2017-04-28 07:55:25 Haw Loeung bug added subscriber Haw Loeung
2020-06-24 02:49:47 Haw Loeung nominated for series Ubuntu Xenial
2020-06-24 02:49:47 Haw Loeung bug task added apache2 (Ubuntu Xenial)
2020-06-24 02:49:58 Haw Loeung bug added subscriber The Canonical Sysadmins
2020-06-24 02:50:01 Haw Loeung apache2 (Ubuntu Xenial): status New Confirmed
2020-06-30 13:37:53 Paride Legovini apache2 (Ubuntu Xenial): status Confirmed Triaged
2020-06-30 13:38:26 Paride Legovini apache2 (Ubuntu Xenial): importance Undecided Low
2020-06-30 13:38:36 Paride Legovini tags server-next
2020-07-19 21:06:22 Bryce Harrington tags server-next
2020-07-19 21:11:19 Bryce Harrington apache2 (Ubuntu Xenial): status Triaged Incomplete
2020-07-20 23:04:52 Bryce Harrington description OS: Ubuntu 14.04 LTS Kernel: 3.13.0-79-generic x86_64 Apache: 2.4.7-1ubuntu4.5 In the default Apache 2.4 config on Ubuntu 14.04 LTS is the following set in /etc/apache2/apache2.conf: Mutex file:${APACHE_LOCK_DIR} default (/debian/config-dir/apache2.conf in http://archive.ubuntu.com/ubuntu/pool/main/a/apache2/apache2_2.4.7-1ubuntu4.5.debian.tar.gz) which leads to the following output of "apache2ctl -t -D DUMP_RUN_CFG": Mutex default: dir="/var/lock/apache2" mechanism=fcntl This leads constantly to a lot of these warning/emergency messages on a server with 200 busy worker threads, 100 Requests/s, 300 KB/s: [Tue Mar 08 16:08:18.596653 2016] [ssl:warn] [pid 8339:tid 140182179256064] (35)Resource deadlock avoided: AH02026: Failed to acquire SSL session cache lock [Wed Mar 09 07:09:31.099331 2016] [mpm_worker:emerg] [pid 26526:tid 139668485949184] (35)Resource deadlock avoided: AH00273: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully. Solution (as suggested by Yann Ylavic from Apache): Commenting (removing) the Mutex directive, which leads to the following output of "apache2ctl -t -D DUMP_RUN_CFG": Mutex default: dir="/var/run/apache2/" mechanism=default Then, there are no error messages anymore. For the discussion, see the corresponding Apache httpd-users mailing list thread: http://httpd.markmail.org/message/c7w5aujfmy2kfazi (thread subject 'Lots of messages "[ssl:warn] Resource deadlock avoided: AH02026: Failed to acquire SSL session cache lock"' from 2016-03-08) Here some more information: # apache2ctl -V Server version: Apache/2.4.7 (Ubuntu) Server built: Jul 24 2015 17:25:11 Server's Module Magic Number: 20120211:27 Server loaded: APR 1.5.1-dev, APR-UTIL 1.5.3 Compiled using: APR 1.5.1-dev, APR-UTIL 1.5.3 Architecture: 64-bit Server MPM: worker threaded: yes (fixed thread count) forked: yes (variable process count) Server compiled with.... -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=256 -D HTTPD_ROOT="/etc/apache2" -D SUEXEC_BIN="/usr/lib/apache2/suexec" -D DEFAULT_PIDLOG="/var/run/apache2.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="mime.types" -D SERVER_CONFIG_FILE="apache2.conf" [Impact] The default apache2.conf causes apache to issue streams of error messages about deadlocks acquiring the SSL session cache lock. Users are still reporting seeing this flaw in production (Xenial-based) hosts. [Test Case] Reproduction steps TBD. Problem exhibits on high load systems. Verification will need to be done by those seeing the issue in production. [Regression Potential] Since this only changes the config installed by default, it won't impact existing installations, however behaviors to watch for would be SSL-related or configuration-related oddnesses. [Fix] Backport a fix applied in bionic and newer, that modifies the makes Apache use pthread mutexes by default on Linux, or fctnl on other architectures that lack robust pthread muxexes. [Original Report] OS: Ubuntu 14.04 LTS Kernel: 3.13.0-79-generic x86_64 Apache: 2.4.7-1ubuntu4.5 In the default Apache 2.4 config on Ubuntu 14.04 LTS is the following set in /etc/apache2/apache2.conf: Mutex file:${APACHE_LOCK_DIR} default (/debian/config-dir/apache2.conf in http://archive.ubuntu.com/ubuntu/pool/main/a/apache2/apache2_2.4.7-1ubuntu4.5.debian.tar.gz) which leads to the following output of "apache2ctl -t -D DUMP_RUN_CFG": Mutex default: dir="/var/lock/apache2" mechanism=fcntl This leads constantly to a lot of these warning/emergency messages on a server with 200 busy worker threads, 100 Requests/s, 300 KB/s: [Tue Mar 08 16:08:18.596653 2016] [ssl:warn] [pid 8339:tid 140182179256064] (35)Resource deadlock avoided: AH02026: Failed to acquire SSL session cache lock [Wed Mar 09 07:09:31.099331 2016] [mpm_worker:emerg] [pid 26526:tid 139668485949184] (35)Resource deadlock avoided: AH00273: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully. Solution (as suggested by Yann Ylavic from Apache): Commenting (removing) the Mutex directive, which leads to the following output of "apache2ctl -t -D DUMP_RUN_CFG": Mutex default: dir="/var/run/apache2/" mechanism=default Then, there are no error messages anymore. For the discussion, see the corresponding Apache httpd-users mailing list thread: http://httpd.markmail.org/message/c7w5aujfmy2kfazi (thread subject 'Lots of messages "[ssl:warn] Resource deadlock avoided: AH02026: Failed to acquire SSL session cache lock"' from 2016-03-08) Here some more information: # apache2ctl -V Server version: Apache/2.4.7 (Ubuntu) Server built: Jul 24 2015 17:25:11 Server's Module Magic Number: 20120211:27 Server loaded: APR 1.5.1-dev, APR-UTIL 1.5.3 Compiled using: APR 1.5.1-dev, APR-UTIL 1.5.3 Architecture: 64-bit Server MPM: worker   threaded: yes (fixed thread count)     forked: yes (variable process count) Server compiled with....  -D APR_HAS_SENDFILE  -D APR_HAS_MMAP  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)  -D APR_USE_SYSVSEM_SERIALIZE  -D APR_USE_PTHREAD_SERIALIZE  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT  -D APR_HAS_OTHER_CHILD  -D AP_HAVE_RELIABLE_PIPED_LOGS  -D DYNAMIC_MODULE_LIMIT=256  -D HTTPD_ROOT="/etc/apache2"  -D SUEXEC_BIN="/usr/lib/apache2/suexec"  -D DEFAULT_PIDLOG="/var/run/apache2.pid"  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"  -D DEFAULT_ERRORLOG="logs/error_log"  -D AP_TYPES_CONFIG_FILE="mime.types"  -D SERVER_CONFIG_FILE="apache2.conf"
2020-07-21 00:07:45 Launchpad Janitor merge proposal linked https://code.launchpad.net/~bryce/ubuntu/+source/apache2/+git/apache2/+merge/387725
2020-07-23 00:48:22 Bryce Harrington description [Impact] The default apache2.conf causes apache to issue streams of error messages about deadlocks acquiring the SSL session cache lock. Users are still reporting seeing this flaw in production (Xenial-based) hosts. [Test Case] Reproduction steps TBD. Problem exhibits on high load systems. Verification will need to be done by those seeing the issue in production. [Regression Potential] Since this only changes the config installed by default, it won't impact existing installations, however behaviors to watch for would be SSL-related or configuration-related oddnesses. [Fix] Backport a fix applied in bionic and newer, that modifies the makes Apache use pthread mutexes by default on Linux, or fctnl on other architectures that lack robust pthread muxexes. [Original Report] OS: Ubuntu 14.04 LTS Kernel: 3.13.0-79-generic x86_64 Apache: 2.4.7-1ubuntu4.5 In the default Apache 2.4 config on Ubuntu 14.04 LTS is the following set in /etc/apache2/apache2.conf: Mutex file:${APACHE_LOCK_DIR} default (/debian/config-dir/apache2.conf in http://archive.ubuntu.com/ubuntu/pool/main/a/apache2/apache2_2.4.7-1ubuntu4.5.debian.tar.gz) which leads to the following output of "apache2ctl -t -D DUMP_RUN_CFG": Mutex default: dir="/var/lock/apache2" mechanism=fcntl This leads constantly to a lot of these warning/emergency messages on a server with 200 busy worker threads, 100 Requests/s, 300 KB/s: [Tue Mar 08 16:08:18.596653 2016] [ssl:warn] [pid 8339:tid 140182179256064] (35)Resource deadlock avoided: AH02026: Failed to acquire SSL session cache lock [Wed Mar 09 07:09:31.099331 2016] [mpm_worker:emerg] [pid 26526:tid 139668485949184] (35)Resource deadlock avoided: AH00273: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully. Solution (as suggested by Yann Ylavic from Apache): Commenting (removing) the Mutex directive, which leads to the following output of "apache2ctl -t -D DUMP_RUN_CFG": Mutex default: dir="/var/run/apache2/" mechanism=default Then, there are no error messages anymore. For the discussion, see the corresponding Apache httpd-users mailing list thread: http://httpd.markmail.org/message/c7w5aujfmy2kfazi (thread subject 'Lots of messages "[ssl:warn] Resource deadlock avoided: AH02026: Failed to acquire SSL session cache lock"' from 2016-03-08) Here some more information: # apache2ctl -V Server version: Apache/2.4.7 (Ubuntu) Server built: Jul 24 2015 17:25:11 Server's Module Magic Number: 20120211:27 Server loaded: APR 1.5.1-dev, APR-UTIL 1.5.3 Compiled using: APR 1.5.1-dev, APR-UTIL 1.5.3 Architecture: 64-bit Server MPM: worker   threaded: yes (fixed thread count)     forked: yes (variable process count) Server compiled with....  -D APR_HAS_SENDFILE  -D APR_HAS_MMAP  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)  -D APR_USE_SYSVSEM_SERIALIZE  -D APR_USE_PTHREAD_SERIALIZE  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT  -D APR_HAS_OTHER_CHILD  -D AP_HAVE_RELIABLE_PIPED_LOGS  -D DYNAMIC_MODULE_LIMIT=256  -D HTTPD_ROOT="/etc/apache2"  -D SUEXEC_BIN="/usr/lib/apache2/suexec"  -D DEFAULT_PIDLOG="/var/run/apache2.pid"  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"  -D DEFAULT_ERRORLOG="logs/error_log"  -D AP_TYPES_CONFIG_FILE="mime.types"  -D SERVER_CONFIG_FILE="apache2.conf" [Impact] The default apache2.conf causes apache to issue streams of error messages about deadlocks acquiring the SSL session cache lock. Users are still reporting seeing this flaw in production (Xenial-based) hosts. [Test Case] Reproduction steps TBD. Problem exhibits on high load systems. Verification will need to be done by those seeing the issue in production. [Regression Potential] Since this only changes the config installed by default, it won't impact existing installations, however behaviors to watch for would be SSL-related or configuration-related oddnesses. [Fix] Backport a fix applied in bionic and newer, that modifies the makes Apache use pthread mutexes by default on Linux, or fctnl on other architectures that lack robust pthread muxexes. [Other Info] Users should be aware that if they haven't changed /etc/apache2/apache2.conf this will automatically apply the fix, but users that have customized apache2.conf may still need to add it manually. [Original Report] OS: Ubuntu 14.04 LTS Kernel: 3.13.0-79-generic x86_64 Apache: 2.4.7-1ubuntu4.5 In the default Apache 2.4 config on Ubuntu 14.04 LTS is the following set in /etc/apache2/apache2.conf: Mutex file:${APACHE_LOCK_DIR} default (/debian/config-dir/apache2.conf in http://archive.ubuntu.com/ubuntu/pool/main/a/apache2/apache2_2.4.7-1ubuntu4.5.debian.tar.gz) which leads to the following output of "apache2ctl -t -D DUMP_RUN_CFG": Mutex default: dir="/var/lock/apache2" mechanism=fcntl This leads constantly to a lot of these warning/emergency messages on a server with 200 busy worker threads, 100 Requests/s, 300 KB/s: [Tue Mar 08 16:08:18.596653 2016] [ssl:warn] [pid 8339:tid 140182179256064] (35)Resource deadlock avoided: AH02026: Failed to acquire SSL session cache lock [Wed Mar 09 07:09:31.099331 2016] [mpm_worker:emerg] [pid 26526:tid 139668485949184] (35)Resource deadlock avoided: AH00273: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully. Solution (as suggested by Yann Ylavic from Apache): Commenting (removing) the Mutex directive, which leads to the following output of "apache2ctl -t -D DUMP_RUN_CFG": Mutex default: dir="/var/run/apache2/" mechanism=default Then, there are no error messages anymore. For the discussion, see the corresponding Apache httpd-users mailing list thread: http://httpd.markmail.org/message/c7w5aujfmy2kfazi (thread subject 'Lots of messages "[ssl:warn] Resource deadlock avoided: AH02026: Failed to acquire SSL session cache lock"' from 2016-03-08) Here some more information: # apache2ctl -V Server version: Apache/2.4.7 (Ubuntu) Server built: Jul 24 2015 17:25:11 Server's Module Magic Number: 20120211:27 Server loaded: APR 1.5.1-dev, APR-UTIL 1.5.3 Compiled using: APR 1.5.1-dev, APR-UTIL 1.5.3 Architecture: 64-bit Server MPM: worker   threaded: yes (fixed thread count)     forked: yes (variable process count) Server compiled with....  -D APR_HAS_SENDFILE  -D APR_HAS_MMAP  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)  -D APR_USE_SYSVSEM_SERIALIZE  -D APR_USE_PTHREAD_SERIALIZE  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT  -D APR_HAS_OTHER_CHILD  -D AP_HAVE_RELIABLE_PIPED_LOGS  -D DYNAMIC_MODULE_LIMIT=256  -D HTTPD_ROOT="/etc/apache2"  -D SUEXEC_BIN="/usr/lib/apache2/suexec"  -D DEFAULT_PIDLOG="/var/run/apache2.pid"  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"  -D DEFAULT_ERRORLOG="logs/error_log"  -D AP_TYPES_CONFIG_FILE="mime.types"  -D SERVER_CONFIG_FILE="apache2.conf"
2020-07-29 14:44:12 Robie Basak apache2 (Ubuntu Xenial): status Incomplete Fix Committed
2020-07-29 14:44:13 Robie Basak bug added subscriber Ubuntu Stable Release Updates Team
2020-07-29 14:44:14 Robie Basak bug added subscriber SRU Verification
2020-07-29 14:44:17 Robie Basak tags verification-needed verification-needed-xenial
2020-07-29 23:00:41 Haw Loeung tags verification-needed verification-needed-xenial verification-done verification-done-xenial
2020-08-13 14:16:59 Launchpad Janitor apache2 (Ubuntu Xenial): status Fix Committed Fix Released
2020-08-13 14:16:59 Launchpad Janitor cve linked 2020-1927
2020-08-13 14:16:59 Launchpad Janitor cve linked 2020-1934