mpm_event fails to restart all children gracefully

[Christian Ehrhardt ]

Reported on the ML [1] and seems to be a valid and real bug for some active releases.

It is in 2.4.49 and thereby fixed in Jammy and later.
Introduced in 2.3.34 so it wasn't present in Bionic yet.

Upstream bug [2]

It first sounded (by the words) a bit like this old bug [3], but it is an entirely new thing.

Since it is fixed upstream we know the change which looks kind of small and reasonable [4].

Maybe someone could evaluate how well (or not) applicable that would be to Focal/Impish?

@Mike - if you read this later, to SRU it we will need a testcase and the upstream bug isn't very helpful. If you would be able to add a step by step description how to trigger the case that would be very helpful.

States from here:
- Triaged for now, action is to eval the applicability of the change
- Then if repro steps exists let us look at an SRU, if not -> incomplete

[1]: https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2022-June/019284.html
[2]: https://bz.apache.org/bugzilla/show_bug.cgi?id=63169
[3]: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1466926
[4]: https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/server/mpm/event/event.c?r1=1893109&r2=1893108&pathrev=1893109

[Bryce Harrington (bryce)]

Attached is the patch upstream backported to the 2.4.x branch. It applies cleanly to bionic and focal's apache2 (with -p4), but jammy already has this fix.

From what I understand, this is an issue that crops up only when apache2 is under intense load, so identifying a reliable test case may be a challenge. However, the fix makes sense and the patch seems reasonable to me.

[Bryce Harrington (bryce)]

[Test case needed]

[Bryce Harrington (bryce)]

A PPA with the patch is available at https://launchpad.net/~bryce/+archive/ubuntu/apache2-fix-lp1979942

As Christian already mentioned above, we need a reproducing test case in order to move forward with an SRU for this package. Or, if an affected person can assist with reproducing the fault on their own system, and then validating the fix we may be able to proceed. In that case we'll need you to also reproduce the issue and fix again later, as part of the final acceptance verification.

Can someone affected by this issue assist? We need either a reproducing test case, or we need someone willing to handle doing the testing themselves.

[Bryce Harrington (bryce)]

Still waiting on a reproducer test case for this bug, before we can move forward deploying the fix.

I took a shot at trying myself using the description from the mailing list, but things seem to work as expected and I did not reproduce the failure using the current apache2 2.4.41-4ubuntu3.12 from focal-updates:

host:> lxc login apache2-fix-lp1979942-focal
$ sudo apt-get update
$ sudo apt-get full-upgrade
$ apt-get install apache2

Restart apache2
$ sudo systemctl restart apache2

Verify things are running okay
$ sudo systemctl status apache2
$ ps aux | grep apache2
root 4029 0.0 0.0 6676 4676 ? Ss 23:09 0:00 /usr/sbin/apache2 -k start
www-data 4030 0.0 0.0 1997996 4384 ? Sl 23:09 0:00 /usr/sbin/apache2 -k start
www-data 4031 0.0 0.0 1997996 4384 ? Sl 23:09 0:00 /usr/sbin/apache2 -k start

$ sudo /etc/init.d/apache2 graceful

Verify some apache processes weren't killed
$ ps aux | grep apache2
root 4029 0.0 0.0 6684 5136 ? Ss 23:09 0:00 /usr/sbin/apache2 -k start
www-data 4122 0.0 0.0 1997996 4388 ? Sl 23:11 0:00 /usr/sbin/apache2 -k start
www-data 4123 0.0 0.0 1997996 4388 ? Sl 23:11 0:00 /usr/sbin/apache2 -k start

Notice here that the root process PID 4029 did not restart, but the two
child www-data ones did.

Verify `apachectl fullstatus` shows state is "stopping... no (old gen)"

Perform logrotate
# ls -l /var/log/apache2/
total 28
-rw-r----- 1 root adm 146 Jan 17 23:13 access.log
-rw-r----- 1 root adm 1092 Jan 17 23:11 error.log
-rw-r----- 1 root adm 411 Jan 17 00:00 error.log.1
-rw-r----- 1 root adm 298 Jan 16 00:00 error.log.2.gz
...
# logrotate -l /tmp/apache-rotate.log /etc/logrotate.conf
# ls -l /var/log/apache2/
total 40
-rw-r----- 1 root adm 0 Jan 17 23:14 access.log
-rw-r----- 1 root adm 146 Jan 17 23:13 access.log.1
-rw-r----- 1 root adm 1506 Jan 17 23:14 error.log
-rw-r----- 1 root adm 411 Jan 17 00:00 error.log.1
-rw-r----- 1 root adm 298 Jan 16 00:00 error.log.2.gz

Verify whether the old, rotated logfiles are still kept open
# lsof | grep /apache2.log
# lsof | grep /error.log
apache2 4029 root 2w REG 253,5 1506 656457 /var/log/apache2/error.log
apache2 4224 www-data 2w REG 253,5 1506 656457 /var/log/apache2/error.log
...

There are no logs still open from the old www-data pids, however the
root process PID 4029 does hold open an error.log

Correct me if I'm wrong, but I'm not spotting the described issue from the above steps. Are there additional steps to configure/setup apache, or workloads to apply that will help trigger it?