Restart policy "Always" doesn't work under certain circumstances
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
open-iscsi (Ubuntu) |
Fix Released
|
High
|
Heitor Alves de Siqueira | ||
Bionic |
Fix Released
|
High
|
Heitor Alves de Siqueira | ||
Cosmic |
Fix Released
|
High
|
Heitor Alves de Siqueira | ||
Disco |
Fix Released
|
High
|
Heitor Alves de Siqueira |
Bug Description
[Impact]
iscsid restart fails when killed with SIGTERM
[Description]
If systemd tries to execute a command that talks to iscsid via iscsid.socket, it can hang if iscsid is not running (or is in the process of being stopped). This can happen due to the current ExecStop= directive, which calls iscsiadm to kill iscsid, and prevents the service from being restarted even when we set Restart=always in the iscsid unit file.
The solution is to let systemd terminate iscsid by itself. The default action
when omitting the ExecStop directive is to send SIGTERM to the process group,
which is equivalent to invoking "iscsiadm -k" (the current ExecStop command).
[Test Case]
1) Deploy a Disco VM e.g. with uvt-kvm
$ uvt-kvm create disco release=disco
2) Run the deploy-tgt.sh script in the VM to setup an iSCSI target in the
localhost. The script will install tgt and open-iscsi, configure a 1G
file-backed iSCSI target, login and restart iscsid.service
ubuntu@disco:~$ ./deploy-tgt.sh
3) Kill iscsid with SIGTERM
ubuntu@disco:~$ sudo pkill iscsid
4) Try to stop iscsid.service and check whether it hangs
ubuntu@disco:~$ sudo systemctl stop iscsid
If we remove the ExecStop= directive, it works as expected:
ubuntu@disco:~$ sudo systemctl stop iscsid
ubuntu@disco:~$
This also causes Restart=always to work as expected.
[Regression Potential]
This shouldn't introduce any regressions, since iscsiadm -k just sends SIGTERM to iscsid's process group and that's equivalent to the default systemd ExecStop action. Nonetheless, changes will be tested with autopkgtests and different iscsi scenarios.
tags: | added: sts-sponsor |
Changed in open-iscsi (Ubuntu Cosmic): | |
status: | Confirmed → In Progress |
Changed in open-iscsi (Ubuntu Bionic): | |
status: | Confirmed → In Progress |
tags: | removed: sts-sponsor |
We triaged this to the fact that ExecStop runs a command that tries to talk to iscsid, and iscsid is socket-activated on Ubuntu, so the command hangs while trying to talk to a socket that systemd is waiting to dispatch to the iscsid service that is in the process of being stopped.
From reading the manpage on iscsiadm, we should just omit the ExecStop command in favor of using the default semantics of KillSignal=SIGTERM.