Activity log for bug #1988224

Date Who What changed Old value New value Message
2022-08-30 20:46:39 Daniel bug added bug
2022-08-31 19:23:24 Sergio Durigan Junior bug watch added https://bz.apache.org/bugzilla/show_bug.cgi?id=65769
2022-08-31 19:23:28 Sergio Durigan Junior apache2 (Ubuntu): status New Incomplete
2022-08-31 19:23:32 Sergio Durigan Junior bug added subscriber Sergio Durigan Junior
2022-09-01 00:06:04 Sergio Durigan Junior bug added subscriber Ubuntu Server
2022-09-01 00:06:07 Sergio Durigan Junior tags apache2 apache2 server-todo
2022-09-01 00:09:37 Sergio Durigan Junior nominated for series Ubuntu Jammy
2022-09-01 00:09:37 Sergio Durigan Junior bug task added apache2 (Ubuntu Jammy)
2022-09-01 00:09:43 Sergio Durigan Junior apache2 (Ubuntu Jammy): status New Triaged
2022-09-01 00:09:45 Sergio Durigan Junior apache2 (Ubuntu Jammy): importance Undecided Medium
2022-09-01 00:09:47 Sergio Durigan Junior apache2 (Ubuntu): status Incomplete Fix Released
2022-09-07 15:13:54 Sergio Durigan Junior apache2 (Ubuntu Jammy): assignee Sergio Durigan Junior (sergiodj)
2022-09-07 15:17:42 Bryce Harrington apache2 (Ubuntu Jammy): assignee Sergio Durigan Junior (sergiodj) Bryce Harrington (bryce)
2022-09-08 04:33:43 Bryce Harrington description Since updating the LTS server version from 20.04 to 22.04, I've had problems with apache 2.4.52 (2.4.52-1ubuntu4.1) in mpm_event mode. The child processes for apache fail to start after a period of time. The webserver is unreachable. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. The error_log says: [Tue Aug 30 12:59:38.451188 2022] [http2:warn] [pid 687247:tid 139925644072832] AH10291: h2_workers: cleanup, 1 idle workers did not exit after 5 seconds. # ps xau |grep apache root 899 0.0 0.4 86712 40116 ? Ss Aug25 0:33 /usr/sbin/apache2 -k start www-data 901 0.0 0.0 3736 156 ? Ss Aug25 0:15 /usr/bin/htcacheclean -d 120 -p /var/cache/apache2/mod_cache_disk -l 300M -n www-data 687242 0.0 0.3 87020 30104 ? S 02:00 0:00 /usr/sbin/apache2 -k start Other processes are gone. The problem is known and already fixed in 2.4.53, see: https://bz.apache.org/bugzilla/show_bug.cgi?id=65769 I haven't found anything that this problem has already been fixed in the ubuntu version of apache. That's why I'm making this bug report. Many Thanks. [Impact] Starting with Apache 2.4.51 the child processes for Apache fail to start after a period of time or after so many requests. The issue became much more frequent with 2.4.52 due to fixes in accounting of active_daemons. Apache will stop accepting new connection until the parent Apache process is eventually restarted. Requests for new web pages will just hang. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. Seems to be affecting mostly servers using the event MPM. Reverting to focal's Apache 2.4.41 or moving to kinetic's 2.4.53 resolves it. [Test Case] $ lxc launch ubuntu:jammy test-apache2 --vm $ lxc shell test-apache2 # apt update && apt dist-upgrade -y # apt install apache2 lynx -y # cat > /etc/apache2/mods-enabled/mpm_event.conf << __EOF__ <IfModule mpm_event_module> StartServers 1 MinSpareThreads 1 MaxSpareThreads 1 ThreadsPerChild 1 MaxRequestWorkers 1 MaxConnectionsPerChild 1 </IfModule> __EOF__ # systemctl restart apache2 # while lynx -dump -read_timeout=10 localhost/server-status; do continue; done ... This command should never return, but it will crash eventually ... [Where Problems Could Occur] TBD [Original Report] Since updating the LTS server version from 20.04 to 22.04, I've had problems with apache 2.4.52 (2.4.52-1ubuntu4.1) in mpm_event mode. The child processes for apache fail to start after a period of time. The webserver is unreachable. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. The error_log says: [Tue Aug 30 12:59:38.451188 2022] [http2:warn] [pid 687247:tid 139925644072832] AH10291: h2_workers: cleanup, 1 idle workers did not exit after 5 seconds. # ps xau |grep apache root 899 0.0 0.4 86712 40116 ? Ss Aug25 0:33 /usr/sbin/apache2 -k start www-data 901 0.0 0.0 3736 156 ? Ss Aug25 0:15 /usr/bin/htcacheclean -d 120 -p /var/cache/apache2/mod_cache_disk -l 300M -n www-data 687242 0.0 0.3 87020 30104 ? S 02:00 0:00 /usr/sbin/apache2 -k start Other processes are gone. The problem is known and already fixed in 2.4.53, see: https://bz.apache.org/bugzilla/show_bug.cgi?id=65769 I haven't found anything that this problem has already been fixed in the ubuntu version of apache. That's why I'm making this bug report. Many Thanks.
2022-09-30 17:45:17 Bryce Harrington description [Impact] Starting with Apache 2.4.51 the child processes for Apache fail to start after a period of time or after so many requests. The issue became much more frequent with 2.4.52 due to fixes in accounting of active_daemons. Apache will stop accepting new connection until the parent Apache process is eventually restarted. Requests for new web pages will just hang. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. Seems to be affecting mostly servers using the event MPM. Reverting to focal's Apache 2.4.41 or moving to kinetic's 2.4.53 resolves it. [Test Case] $ lxc launch ubuntu:jammy test-apache2 --vm $ lxc shell test-apache2 # apt update && apt dist-upgrade -y # apt install apache2 lynx -y # cat > /etc/apache2/mods-enabled/mpm_event.conf << __EOF__ <IfModule mpm_event_module> StartServers 1 MinSpareThreads 1 MaxSpareThreads 1 ThreadsPerChild 1 MaxRequestWorkers 1 MaxConnectionsPerChild 1 </IfModule> __EOF__ # systemctl restart apache2 # while lynx -dump -read_timeout=10 localhost/server-status; do continue; done ... This command should never return, but it will crash eventually ... [Where Problems Could Occur] TBD [Original Report] Since updating the LTS server version from 20.04 to 22.04, I've had problems with apache 2.4.52 (2.4.52-1ubuntu4.1) in mpm_event mode. The child processes for apache fail to start after a period of time. The webserver is unreachable. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. The error_log says: [Tue Aug 30 12:59:38.451188 2022] [http2:warn] [pid 687247:tid 139925644072832] AH10291: h2_workers: cleanup, 1 idle workers did not exit after 5 seconds. # ps xau |grep apache root 899 0.0 0.4 86712 40116 ? Ss Aug25 0:33 /usr/sbin/apache2 -k start www-data 901 0.0 0.0 3736 156 ? Ss Aug25 0:15 /usr/bin/htcacheclean -d 120 -p /var/cache/apache2/mod_cache_disk -l 300M -n www-data 687242 0.0 0.3 87020 30104 ? S 02:00 0:00 /usr/sbin/apache2 -k start Other processes are gone. The problem is known and already fixed in 2.4.53, see: https://bz.apache.org/bugzilla/show_bug.cgi?id=65769 I haven't found anything that this problem has already been fixed in the ubuntu version of apache. That's why I'm making this bug report. Many Thanks. [Impact] Starting with Apache 2.4.51 the child processes for Apache fail to start after a period of time or after so many requests. The issue became much more frequent with 2.4.52 due to fixes in accounting of active_daemons. Apache will stop accepting new connection until the parent Apache process is eventually restarted. Requests for new web pages will just hang. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. Seems to be affecting mostly servers using the event MPM. Reverting to focal's Apache 2.4.41 or moving to kinetic's 2.4.53 resolves it. [Test Case] $ lxc launch ubuntu:jammy apache2-sru-lp1988224-jammy --vm $ lxc shell apache2-sru-lp1988224-jammy # apt update && apt dist-upgrade -y # apt install apache2 lynx -y # cat > /etc/apache2/mods-enabled/mpm_event.conf << __EOF__ <IfModule mpm_event_module>     StartServers 1     MinSpareThreads 1     MaxSpareThreads 1     ThreadsPerChild 1     MaxRequestWorkers 1     MaxConnectionsPerChild 1 </IfModule> __EOF__ # systemctl restart apache2 # while lynx -dump -read_timeout=10 localhost/server-status; do continue; done ... This command should never return, but it will crash eventually ... [Where Problems Could Occur] TBD [Original Report] Since updating the LTS server version from 20.04 to 22.04, I've had problems with apache 2.4.52 (2.4.52-1ubuntu4.1) in mpm_event mode. The child processes for apache fail to start after a period of time. The webserver is unreachable. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. The error_log says: [Tue Aug 30 12:59:38.451188 2022] [http2:warn] [pid 687247:tid 139925644072832] AH10291: h2_workers: cleanup, 1 idle workers did not exit after 5 seconds. # ps xau |grep apache root 899 0.0 0.4 86712 40116 ? Ss Aug25 0:33 /usr/sbin/apache2 -k start www-data 901 0.0 0.0 3736 156 ? Ss Aug25 0:15 /usr/bin/htcacheclean -d 120 -p /var/cache/apache2/mod_cache_disk -l 300M -n www-data 687242 0.0 0.3 87020 30104 ? S 02:00 0:00 /usr/sbin/apache2 -k start Other processes are gone. The problem is known and already fixed in 2.4.53, see: https://bz.apache.org/bugzilla/show_bug.cgi?id=65769 I haven't found anything that this problem has already been fixed in the ubuntu version of apache. That's why I'm making this bug report. Many Thanks.
2022-09-30 17:58:26 Bryce Harrington description [Impact] Starting with Apache 2.4.51 the child processes for Apache fail to start after a period of time or after so many requests. The issue became much more frequent with 2.4.52 due to fixes in accounting of active_daemons. Apache will stop accepting new connection until the parent Apache process is eventually restarted. Requests for new web pages will just hang. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. Seems to be affecting mostly servers using the event MPM. Reverting to focal's Apache 2.4.41 or moving to kinetic's 2.4.53 resolves it. [Test Case] $ lxc launch ubuntu:jammy apache2-sru-lp1988224-jammy --vm $ lxc shell apache2-sru-lp1988224-jammy # apt update && apt dist-upgrade -y # apt install apache2 lynx -y # cat > /etc/apache2/mods-enabled/mpm_event.conf << __EOF__ <IfModule mpm_event_module>     StartServers 1     MinSpareThreads 1     MaxSpareThreads 1     ThreadsPerChild 1     MaxRequestWorkers 1     MaxConnectionsPerChild 1 </IfModule> __EOF__ # systemctl restart apache2 # while lynx -dump -read_timeout=10 localhost/server-status; do continue; done ... This command should never return, but it will crash eventually ... [Where Problems Could Occur] TBD [Original Report] Since updating the LTS server version from 20.04 to 22.04, I've had problems with apache 2.4.52 (2.4.52-1ubuntu4.1) in mpm_event mode. The child processes for apache fail to start after a period of time. The webserver is unreachable. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. The error_log says: [Tue Aug 30 12:59:38.451188 2022] [http2:warn] [pid 687247:tid 139925644072832] AH10291: h2_workers: cleanup, 1 idle workers did not exit after 5 seconds. # ps xau |grep apache root 899 0.0 0.4 86712 40116 ? Ss Aug25 0:33 /usr/sbin/apache2 -k start www-data 901 0.0 0.0 3736 156 ? Ss Aug25 0:15 /usr/bin/htcacheclean -d 120 -p /var/cache/apache2/mod_cache_disk -l 300M -n www-data 687242 0.0 0.3 87020 30104 ? S 02:00 0:00 /usr/sbin/apache2 -k start Other processes are gone. The problem is known and already fixed in 2.4.53, see: https://bz.apache.org/bugzilla/show_bug.cgi?id=65769 I haven't found anything that this problem has already been fixed in the ubuntu version of apache. That's why I'm making this bug report. Many Thanks. [Impact] Starting with Apache 2.4.51 the child processes for Apache fail to start after a period of time or after so many requests. The issue became much more frequent with 2.4.52 due to fixes in accounting of active_daemons. Apache will stop accepting new connection until the parent Apache process is eventually restarted. Requests for new web pages will just hang. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. Seems to be affecting mostly servers using the event MPM. Reverting to focal's Apache 2.4.41 or moving to kinetic's 2.4.53 resolves it. [Test Case] $ lxc launch ubuntu:jammy apache2-sru-lp1988224-jammy --vm $ lxc shell apache2-sru-lp1988224-jammy # apt update && apt dist-upgrade -y # apt install apache2 lynx -y # cat > /etc/apache2/mods-enabled/mpm_event.conf << __EOF__ <IfModule mpm_event_module>     StartServers 1     MinSpareThreads 1     MaxSpareThreads 1     ThreadsPerChild 1     MaxRequestWorkers 1     MaxConnectionsPerChild 1 </IfModule> __EOF__ # systemctl restart apache2 # while lynx -dump -read_timeout=10 localhost/server-status; do continue; done ... This command should never return, but it will crash eventually ... [Where Problems Could Occur] The included patches involve changes to connection behavior, so it would be worth watching for reports of misbehaviors relating to client processes such as not loading pages from the webserver. The patches change C code, so the usual sorts of regression risks apply. These don't change memory management or pointer behavior, so its less likely that regressions would involve memory leaks or invalid pointers, but more likely to be loop misbehaviors (getting stuck, skipping an operation or doing it too many times, etc.) The upstream codebase does not have further refinements to these patches in particular, but there are subsequent patches to the files in question for unrelated issues. It doesn't look like we require any of those for this problem, but it's conceivable something is overlooked. If this is true, then obviously look for problem reports that match description of one of those upstream commits. [Original Report] Since updating the LTS server version from 20.04 to 22.04, I've had problems with apache 2.4.52 (2.4.52-1ubuntu4.1) in mpm_event mode. The child processes for apache fail to start after a period of time. The webserver is unreachable. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. The error_log says: [Tue Aug 30 12:59:38.451188 2022] [http2:warn] [pid 687247:tid 139925644072832] AH10291: h2_workers: cleanup, 1 idle workers did not exit after 5 seconds. # ps xau |grep apache root 899 0.0 0.4 86712 40116 ? Ss Aug25 0:33 /usr/sbin/apache2 -k start www-data 901 0.0 0.0 3736 156 ? Ss Aug25 0:15 /usr/bin/htcacheclean -d 120 -p /var/cache/apache2/mod_cache_disk -l 300M -n www-data 687242 0.0 0.3 87020 30104 ? S 02:00 0:00 /usr/sbin/apache2 -k start Other processes are gone. The problem is known and already fixed in 2.4.53, see: https://bz.apache.org/bugzilla/show_bug.cgi?id=65769 I haven't found anything that this problem has already been fixed in the ubuntu version of apache. That's why I'm making this bug report. Many Thanks.
2022-10-03 20:15:52 Bryce Harrington description [Impact] Starting with Apache 2.4.51 the child processes for Apache fail to start after a period of time or after so many requests. The issue became much more frequent with 2.4.52 due to fixes in accounting of active_daemons. Apache will stop accepting new connection until the parent Apache process is eventually restarted. Requests for new web pages will just hang. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. Seems to be affecting mostly servers using the event MPM. Reverting to focal's Apache 2.4.41 or moving to kinetic's 2.4.53 resolves it. [Test Case] $ lxc launch ubuntu:jammy apache2-sru-lp1988224-jammy --vm $ lxc shell apache2-sru-lp1988224-jammy # apt update && apt dist-upgrade -y # apt install apache2 lynx -y # cat > /etc/apache2/mods-enabled/mpm_event.conf << __EOF__ <IfModule mpm_event_module>     StartServers 1     MinSpareThreads 1     MaxSpareThreads 1     ThreadsPerChild 1     MaxRequestWorkers 1     MaxConnectionsPerChild 1 </IfModule> __EOF__ # systemctl restart apache2 # while lynx -dump -read_timeout=10 localhost/server-status; do continue; done ... This command should never return, but it will crash eventually ... [Where Problems Could Occur] The included patches involve changes to connection behavior, so it would be worth watching for reports of misbehaviors relating to client processes such as not loading pages from the webserver. The patches change C code, so the usual sorts of regression risks apply. These don't change memory management or pointer behavior, so its less likely that regressions would involve memory leaks or invalid pointers, but more likely to be loop misbehaviors (getting stuck, skipping an operation or doing it too many times, etc.) The upstream codebase does not have further refinements to these patches in particular, but there are subsequent patches to the files in question for unrelated issues. It doesn't look like we require any of those for this problem, but it's conceivable something is overlooked. If this is true, then obviously look for problem reports that match description of one of those upstream commits. [Original Report] Since updating the LTS server version from 20.04 to 22.04, I've had problems with apache 2.4.52 (2.4.52-1ubuntu4.1) in mpm_event mode. The child processes for apache fail to start after a period of time. The webserver is unreachable. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. The error_log says: [Tue Aug 30 12:59:38.451188 2022] [http2:warn] [pid 687247:tid 139925644072832] AH10291: h2_workers: cleanup, 1 idle workers did not exit after 5 seconds. # ps xau |grep apache root 899 0.0 0.4 86712 40116 ? Ss Aug25 0:33 /usr/sbin/apache2 -k start www-data 901 0.0 0.0 3736 156 ? Ss Aug25 0:15 /usr/bin/htcacheclean -d 120 -p /var/cache/apache2/mod_cache_disk -l 300M -n www-data 687242 0.0 0.3 87020 30104 ? S 02:00 0:00 /usr/sbin/apache2 -k start Other processes are gone. The problem is known and already fixed in 2.4.53, see: https://bz.apache.org/bugzilla/show_bug.cgi?id=65769 I haven't found anything that this problem has already been fixed in the ubuntu version of apache. That's why I'm making this bug report. Many Thanks. [Impact] Starting with Apache 2.4.51 the child processes for Apache fail to start after a period of time or after so many requests. The issue became much more frequent with 2.4.52 due to fixes in accounting of active_daemons. Apache will stop accepting new connection until the parent Apache process is eventually restarted. Requests for new web pages will just hang. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. Seems to be affecting mostly servers using the event MPM. Reverting to focal's Apache 2.4.41 or moving to kinetic's 2.4.53 resolves it. [Test Case] $ lxc launch ubuntu:jammy apache2-sru-lp1988224-jammy --vm $ lxc shell apache2-sru-lp1988224-jammy # apt update && apt dist-upgrade -y # apt install apache2 lynx -y # cat > /etc/apache2/mods-enabled/mpm_event.conf << __EOF__ <IfModule mpm_event_module>     StartServers 1     MinSpareThreads 1     MaxSpareThreads 1     ThreadsPerChild 1     MaxRequestWorkers 1     MaxConnectionsPerChild 1 </IfModule> __EOF__ # systemctl restart apache2 # while lynx -dump -read_timeout=10 localhost/server-status; do continue; done ... This command should never return, but it will crash eventually ... Example failed output: root@apache2-sru-lp1988224-jammy:~# time (while lynx -dump -read_timeout=10 localhost/server-status; do continue; done) Looking up localhost Making HTTP connection to localhost Sending HTTP request. HTTP request sent; waiting for response. Alert!: Socket read failed (too many tries). Connection interrupted. lynx: Can't access startfile http://localhost/server-status real 0m10.082s user 0m0.044s sys 0m0.011s Example of successful output: Srv PID Acc M CPU SS Req Dur Conn Child Slot Client Protocol VHost Request 0-0 5891 1/0/36 W 0.00 0 0 25 0.0 0.00 0.14 127.0.0.1 http/1.1 apache2-sru-lp1988224-jammy.lxd GET /server-status HTTP/1.0 1-0 - 0/0/36 . 0.00 0 0 26 0.0 0.00 0.14 127.0.0.1 http/1.1 apache2-sru-lp1988224-jammy.lxd GET /server-status HTTP/1.0 ... (continuous output) ... [Where Problems Could Occur] The included patches involve changes to connection behavior, so it would be worth watching for reports of misbehaviors relating to client processes such as not loading pages from the webserver. The patches change C code, so the usual sorts of regression risks apply. These don't change memory management or pointer behavior, so its less likely that regressions would involve memory leaks or invalid pointers, but more likely to be loop misbehaviors (getting stuck, skipping an operation or doing it too many times, etc.) The upstream codebase does not have further refinements to these patches in particular, but there are subsequent patches to the files in question for unrelated issues. It doesn't look like we require any of those for this problem, but it's conceivable something is overlooked. If this is true, then obviously look for problem reports that match description of one of those upstream commits. [Original Report] Since updating the LTS server version from 20.04 to 22.04, I've had problems with apache 2.4.52 (2.4.52-1ubuntu4.1) in mpm_event mode. The child processes for apache fail to start after a period of time. The webserver is unreachable. In netstat there are many CLOSE_WAIT and ESTABLISHED entries. The error_log says: [Tue Aug 30 12:59:38.451188 2022] [http2:warn] [pid 687247:tid 139925644072832] AH10291: h2_workers: cleanup, 1 idle workers did not exit after 5 seconds. # ps xau |grep apache root 899 0.0 0.4 86712 40116 ? Ss Aug25 0:33 /usr/sbin/apache2 -k start www-data 901 0.0 0.0 3736 156 ? Ss Aug25 0:15 /usr/bin/htcacheclean -d 120 -p /var/cache/apache2/mod_cache_disk -l 300M -n www-data 687242 0.0 0.3 87020 30104 ? S 02:00 0:00 /usr/sbin/apache2 -k start Other processes are gone. The problem is known and already fixed in 2.4.53, see: https://bz.apache.org/bugzilla/show_bug.cgi?id=65769 I haven't found anything that this problem has already been fixed in the ubuntu version of apache. That's why I'm making this bug report. Many Thanks.
2022-10-03 20:20:17 Launchpad Janitor merge proposal linked https://code.launchpad.net/~bryce/ubuntu/+source/apache2/+git/apache2/+merge/430928
2022-10-04 19:25:30 Bryce Harrington apache2 (Ubuntu Jammy): status Triaged In Progress
2022-10-17 13:28:04 Timo Aaltonen apache2 (Ubuntu Jammy): status In Progress Fix Committed
2022-10-17 13:28:05 Timo Aaltonen bug added subscriber Ubuntu Stable Release Updates Team
2022-10-17 13:28:07 Timo Aaltonen bug added subscriber SRU Verification
2022-10-17 13:28:12 Timo Aaltonen tags apache2 server-todo apache2 server-todo verification-needed verification-needed-jammy
2022-11-10 01:32:57 Bryce Harrington tags apache2 server-todo verification-needed verification-needed-jammy apache2 server-todo verification-done verification-done-jammy
2022-11-16 01:15:03 Launchpad Janitor apache2 (Ubuntu Jammy): status Fix Committed Fix Released
2022-11-16 01:15:09 Brian Murray removed subscriber Ubuntu Stable Release Updates Team