Processes keep running after "systemctl stop"

[Mario Emmenlauer]

I have a number of Ubuntu 22.04 installations on various computers. In the last month, I have seen more and more log messages like the following from different systems:

```
systemd[1]: apache2.service: Unit process 70257 (apache2) remains running after unit stopped.

systemd[1]: exim4.service: Unit process 61890 (exim4) remains running after unit stopped.
```

I started considering this a problem when subsequently, some php processes continued running after `systemctl stop apache2.service`. This can be a problem when rogue processes interfere with my file-based backup. I had at least one case where a rogue `php` process started misbehaving after both `apache2` and `mysql` where stopped. The `php` processes continued running, possibly inside a rogue `apache2` process, but failed to continue its normal operation due to the absence of `mysql`.

Motivation: I have a nightly backup script that stops common services like `exim4`, `apache2` and `mysql`, then performs file-based backup, and subsequently starts services again. I was relying on the fact that processes would end after `systemctl stop`, and do not modify the file system anymore. But this seems not always to be the case?

I do not necessarily think that this is a duplicate of https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2013543. The way I understand the other bug is that systemd forgets about running processes. Here in my case, systemd seems well aware that not all processes ended, at least this is what the log message indicates. But systemd does not force them to quit.

Expected behavior: After `systemctl stop`, I would need all processes to end. This would be required so that I can be sure that it is safe to perform file based backup, without modifications on the file system.

[Mario Emmenlauer (emmenlau)]

I'm running systemd:amd64 249.11-0ubuntu3.10 from jammy-updates on Ubuntu 22.04.3 LTS.

[Mario Emmenlauer (emmenlau)]

This seems to be a related problem:

Sep 29 07:00:30 hyades systemd[1]: apt-daily-upgrade.service: Unit process 51088 (exim4) remains running after unit stopped.

I would think that apt-daily-upgrade relies on exim4 getting actually restarted after an update. If the process remains running, does that not potentially cause trouble?