systemd not respecting EXTEND_TIMEOUT_USEC when stopping service

[Adrian Barkus]

It seems like systemd is not consistently respecting EXTEND_TIMEOUT_USEC. When I stop a service, it is being killed prematurely even after EXTEND_TIMEOUT_USEC was recently sent to systemd (however it is not killed immediately after the default TimeoutStopSec either).

The service `ihm-eco-sputnik-agent.service` includes the following configuration:

  [Service]
  Restart=always
  RestartSec=5
  Type=notify
  TimeoutStartSec=10
  WatchdogSec=60
  TimeoutStopSec=60
  KillMode=mixed
  ...

When I `systemctl restart ihm-eco-sputnik-agent.service`, I see the following logs in `journalctl`:

```
Aug 12 15:43:59 00044bcbe9bc systemd[1]: ihm-eco-sputnik-agent.service: Trying to enqueue job ihm-eco-sputnik-agent.service/restart/replace
Aug 12 15:43:59 00044bcbe9bc systemd[1]: ihm-eco-sputnik-agent.service: Installed new job ihm-eco-sputnik-agent.service/restart as 1149303
Aug 12 15:43:59 00044bcbe9bc systemd[1]: ihm-eco-sputnik-agent.service: Enqueued job ihm-eco-sputnik-agent.service/restart as 1149303
Aug 12 15:43:59 00044bcbe9bc systemd[1]: ihm-eco-sputnik-agent.service: Changed running -> stop-sigterm
Aug 12 15:43:59 00044bcbe9bc systemd[1]: Stopping ihm-eco
sputnik-agent...
-- Subject: Unit ihm-eco-sputnik-agent.service has begun shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit ihm-eco-sputnik-agent.service has begun shutting down.
Aug 12 15:44:49 00044bcbe9bc systemd[1]: ihm-eco-sputnik-agent.service: Got notification message from PID 19520 (WATCHDOG=1)
Aug 12 15:44:49 00044bcbe9bc systemd[1]: ihm-eco-sputnik-agent.service: Got notification message from PID 19520 (EXTEND_TIMEOUT_USEC=60000000)
Aug 12 15:45:09 00044bcbe9bc systemd[1]: ihm-eco-sputnik-agent.service: Got notification message from PID 19520 (WATCHDOG=1)
Aug 12 15:45:09 00044bcbe9bc systemd[1]: ihm-eco-sputnik-agent.service: Got notification message from PID 19520 (EXTEND_TIMEOUT_USEC=60000000)
Aug 12 15:45:10 00044bcbe9bc systemd[1]: ihm-eco-sputnik-agent.service: State 'stop-sigterm' timed out. Killing.
Aug 12 15:45:10 00044bcbe9bc systemd[1]: ihm-eco-sputnik-agent.service: Killing process 19520 (ihm-eco-sputnik) with signal SIGKILL.
Aug 12 15:45:10 00044bcbe9bc systemd[1]: ihm-eco-sputnik-agent.service: Killing process 21114 (systemctl) with signal SIGKILL.
Aug 12 15:45:10 00044bcbe9bc systemd[1]: ihm-eco-sputnik-agent.service: Killing process 21111 (200_IhmEcoDeplo) with signal SIGKILL.
Aug 12 15:45:10 00044bcbe9bc systemd[1]: ihm-eco-sputnik-agent.service: Main process exited, code=killed, status=9/KILL
```

I'm not sure if this is related but I'm also seeing the application successfully writing notification messages which systemd apparently doesn't receive (they aren't logged).

lsb_release -rd:
```
Description: Ubuntu 18.04.5 LTS
Release: 18.04
```

apt-cache policy systemd:
```
systemd:
  Installed: 237-3ubuntu10.42
  Candidate: 237-3ubuntu10.42
  Version table:
 *** 237-3ubuntu10.42 500
        500 s3://ihm-eco-apt-repository-us-west-2.s3-accelerate.amazonaws.com/ihm-eco-apt-repository-us-west-2 bionic-20200810/ubuntu arm64 Packages
        100 /var/lib/dpkg/status
```

[Dan Streetman (ddstreet)]

can you boot with the kernel param 'systemd.log_level=debug' and recreate the problem, and then provide the full journal logs, e.g.:

$ journalctl -b > /tmp/lp1891358.log

[Launchpad Janitor (janitor)]

[Expired for systemd (Ubuntu) because there has been no activity for 60 days.]