smartmontools.service: start operation timed out. Terminating.

[andrew bezella]

the smartmontools.service is seen by systemd as timing out and is terminated.

`systemctl status smartmontools.service` reports:

● smartmontools.service - Self Monitoring and Reporting Technology (SMART) Daemon
     Loaded: loaded (/lib/systemd/system/smartmontools.service; enabled; vendor preset: enabled)
     Active: failed (Result: timeout) since Fri 2020-07-17 17:32:59 UTC; 29min ago
       Docs: man:smartd(8)
             man:smartd.conf(5)
    Process: 57674 ExecStart=/usr/sbin/smartd -n $smartd_opts (code=killed, signal=TERM)
   Main PID: 57674 (code=killed, signal=TERM)
     Status: "Checking 38 devices ..."

Jul 17 17:31:38 ia804105.us.archive.org smartd[57674]: Device: /dev/sdal [SAT], opened
Jul 17 17:31:38 ia804105.us.archive.org smartd[57674]: Device: /dev/sdal [SAT], WDC WD4005FZBX-00K5WB0, S/N:VBG7B3WR, WWN:5-000cca-095c3585c, FW:01.01A01, 4.00 TB
Jul 17 17:31:38 ia804105.us.archive.org smartd[57674]: Device: /dev/sdal [SAT], not found in smartd database.
Jul 17 17:31:38 ia804105.us.archive.org smartd[57674]: Device: /dev/sdal [SAT], enabled SMART Attribute Autosave.
Jul 17 17:31:38 ia804105.us.archive.org smartd[57674]: Device: /dev/sdal [SAT], enabled SMART Automatic Offline Testing.
Jul 17 17:31:38 ia804105.us.archive.org smartd[57674]: Device: /dev/sdal [SAT], is SMART capable. Adding to "monitor" list.
Jul 17 17:31:38 ia804105.us.archive.org smartd[57674]: Monitoring 38 ATA/SATA, 0 SCSI/SAS and 0 NVMe devices
Jul 17 17:32:59 ia804105.us.archive.org systemd[1]: smartmontools.service: start operation timed out. Terminating.
Jul 17 17:32:59 ia804105.us.archive.org systemd[1]: smartmontools.service: Failed with result 'timeout'.
Jul 17 17:32:59 ia804105.us.archive.org systemd[1]: Failed to start Self Monitoring and Reporting Technology (SMART) Daemon.

per the suggestion from the mailing list thread at https://sourceforge.net/p/smartmontools/mailman/message/33725719/ and the referenced upstream service file at https://www.smartmontools.org/export/4048/trunk/smartmontools/smartd.service.in i used an override file to switch to Type=simple (the default afaict) and that allows the service to start.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: smartmontools 7.1-1build1
ProcVersionSignature: Ubuntu 5.4.0-40.44-generic 5.4.44
Uname: Linux 5.4.0-40-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.3
Architecture: amd64
CasperMD5CheckResult: skip
Date: Fri Jul 17 11:50:19 2020
ProcEnviron:
 SHELL=/bin/zsh
 LANG=en_US.UTF-8
 TERM=rxvt-unicode-256color
 XDG_RUNTIME_DIR=<set>
 PATH=(custom, user)
SourcePackage: smartmontools
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.smartd.conf:
 DEVICESCAN \
     -d sat -d removable -H -l error -l selftest -f -t \
     -n standby -a -o on -S on \
     -s (O/../.././(00|06|12|18)|S/../.././06|L/../../2/06)
mtime.conffile..etc.default.smartmontools: 2020-07-17T10:29:30.610929
mtime.conffile..etc.smartd.conf: 2020-07-17T10:29:29.786929

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "smartmontools_service.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Christian Franke (christian-franke)]

> Status: "Checking 38 devices ..."
> ...
> Jul 17 17:31:38 ia804105.us.archive.org smartd[57674]:
> Monitoring 38 ATA/SATA, 0 SCSI/SAS and 0 NVMe devices
> Jul 17 17:32:59 ia804105.us.archive.org systemd[1]:
> smartmontools.service: start operation timed out. Terminating.

This suggests that the first device checks last longer than the systemd setting DefaultTimeoutStartSec=90s. The current version of smartd notifies READY=1 only after the first checks are completed. In traditional daemon mode (no -n option and no NOTIFY_SOCKET) it would fork() then.

Please try this in smartmontools.service file:
...
 [Service]
 Type=notify
+TimeoutStartSec=300s
...

To find a reasonable TimeoutStartSec value and see which device checks last very long, use for example:

# time /usr/sbin/smartd -q onecheck

Christian Franke
smartmontools.org

[Bryce Harrington (bryce)]

smartmontools_service.patch is a work around rather than a fix we can carry in the package, so I've unmarked it as a solution. The 300s timeout that Christian suggests would be a better workaround.

It would be useful to know whether the delay is simply due to the number of devices (38 it looks like?) versus if there is an issue with one of the devices specifically. Let us know how the analysis goes.

[andrew bezella (abezella)]

thank you for the clarification and the suggestion. i was confused by the timeout occurring after the message "Monitoring 38 ATA/SATA, 0 SCSI/SAS and 0 NVMe devices" as that seemed to indicate a successful startup.

it appears to have been just the number of devices, not an issue with any particular one. raising the timeout gave `smartd` time to start properly.

thanks again, this report can be closed as not-a-bug.