smartd spindown check fails on newer disks, disabling check. Disks then spin-up when polled for SMART data.

[Andrew]

Problem:

Support for additional power modes in modern disks like Seagate ST8000AS0002 causes reporting of some power states as "unknown". Where smartd checks disk while these additional states are active, it detects that the disk fails to comply with ATA standard and ceases to use the -n directive. From then on, smartd will check the disk regardless of current power mode, causing the disk to wake from idle every time smartd polls the disk state.

Symptoms:

* Initially, disks do not wake from standby as smartd detects that they are idle.
* At some point, smartd polls disk when in "unknown" power state. The following appears in /var/log/syslog:
  "Device: <DISK>, CHECK POWER STATUS returned 130, not ATA compliant, ignoring -n Directive" OR
  "Device: <DISK>, CHECK POWER STATUS returned 129, not ATA compliant, ignoring -n Directive"
* Following this, disk wakes from standby (spins up) every time smartd polls for SMART status.

What should have happened:

Support for more recent version of ATA spec for power modes is included in updated smartmontools. By updating, smartmontools will not detect more recent disks as not supporting ATA specification and as such, will not disable -n directive. As such, smartmontools will continue to check if disks are spun down before polling for SMART data.

--> As an aside, and more of an upstream issue, but a better check for smartmon to perform would be "is the disk in known sleep/standby power state" directly before waking it every time, rather than detecting if the disk is ever in an unknown power state and disabling the check. This has the advantage of not breaking the "do not wake from sleep to poll for SMART data" function if further power modes are added to the ATA spec in future revisions as long as no new sleep/standby modes are added...

Cause:

Current version of smartd is r4214. Support for ACS-2 disks not included.
See smartmontools ticket here: https://www.smartmontools.org/ticket/184
Patch included 27/3/16 in r4255 (smartd) and r4256 (smartctl). (About two weeks ago).

Ubuntu version:

Description: Ubuntu Xenial Xerus (development branch)
Release: 16.04

Smartmontools version:

smartmontools:
  Installed: 6.4+svn4214-1
  Candidate: 6.4+svn4214-1
  Version table:
 *** 6.4+svn4214-1 500
        500 http://au.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status

[Christian Franke (christian-franke)]

Fix is included in smartmontools-6.5 release.

[Andrew (caeci11iusad1)]

Is there any likelihood that smartmontools 6.5 will be packaged up and distributed to 16.04, or will I need to wait for 16.10 for a fix for this issue?

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

See https://wiki.ubuntu.com/StableReleaseUpdates for policy, rationale and procedure for updating 16.04 in Ubuntu. I think we can probably do a backport for this, but somebody will need to prepare that and carefully review it for regression risk.

[Andrew (caeci11iusad1)]

That's great news Robbie. While of course there is a risk of regression with any change, doing this as a backport will ensure it's only installed intentionally by people who know why they want it. And by having this available, I'll be able to enable automatic SMART monitoring for my Seagate archive disks (without having them permanently awake, which they're not designed to handle), which will give me some warning if they're on the road to failure. Given I've had several disks fail due to age in recent years (all characterised by increases in pre-failure SMART attributes), I consider this a pretty important thing to have working.

[Nish Aravamudan (nacc)]

Just as an FYI, smartmontools 6.5 is not in Debian or 16.10. In fact, 16.04, 16.10, stretch & sid all have the same version of smartmontools.

I have submitted some test builds that backport the upstream fixes: https://launchpad.net/~nacc/+archive/ubuntu/lp1569843. Please test those to see if they resolve the issue for you.

Based upon the upstream commit, I *think* that existing support is not modified in those changes, so the risk of regression should be low. Can you help confirm that, @caeci11iusad1?

[Andrew (caeci11iusad1)]

Installed and tested over the last few hours...looks good so far. Detected IDLE_B mode instead of an unknown power state of the disk when HDPARM was reporting UNKNOWN state (it also doesn't support the new modes) and a SMART check triggered.

My older Seagate Barracuda 7200.14s (x4) and my single remaining Seagate Barracuda Green seem perfectly happy with this build as well.

Thanks for your help with this Nish :-).

Abbreviated syslog below:

Jul 9 11:56:50 yggdrasil systemd[1]: Started Self Monitoring and Reporting Technology (SMART) Daemon.
Jul 9 11:56:50 yggdrasil smartd[20542]: smartd 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-22-generic] (local build)
Jul 9 11:56:50 yggdrasil smartd[20542]: Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
Jul 9 11:56:50 yggdrasil smartd[20542]: Opened configuration file /etc/smartd.conf
Jul 9 11:56:50 yggdrasil smartd[20542]: Drive: /dev/disk/by-id/ata-FM-25S2S-120GBP2_GS011334308M161641, implied '-a' Directive on line 169 of file /etc/smartd.conf
Jul 9 11:56:50 yggdrasil smartd[20542]: Configuration file /etc/smartd.conf parsed.
…
Jul 9 11:56:50 yggdrasil smartd[20542]: Device: /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408DQT [SAT], opened
Jul 9 11:56:50 yggdrasil smartd[20542]: Device: /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408DQT [SAT], ST8000AS0002-1NA17Z, S/N:Z8408DQT, WWN:5-000c50-086e33fa6, FW:AR13, 8.00 TB
Jul 9 11:56:50 yggdrasil smartd[20542]: Device: /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408DQT [SAT], found in smartd database: Seagate Archive HDD
Jul 9 11:57:00 yggdrasil smartd[20542]: Device: /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408DQT [SAT], ATA settings applied: Standby:242
Jul 9 11:57:00 yggdrasil smartd[20542]: Device: /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408DQT [SAT], is SMART capable. Adding to "monitor" list.
Jul 9 11:57:00 yggdrasil smartd[20542]: Device: /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408DQT [SAT], state read from /var/lib/smartmontools/smartd.ST8000AS0002_1NA17Z-Z8408DQT.ata.state
Jul 9 11:57:00 yggdrasil smartd[20542]: Device: /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408TYD [SAT], opened
Jul 9 11:57:00 yggdrasil smartd[20542]: Device: /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408TYD [SAT], ST8000AS0002-1NA17Z, S/N:Z8408TYD, WWN:5-000c50-086e3b71c, FW:AR13, 8.00 TB
Jul 9 11:57:00 yggdrasil smartd[20542]: Device: /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408TYD [SAT], found in smartd database: Seagate Archive HDD
Jul 9 11:57:11 yggdrasil smartd[20542]: Device: /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408TYD [SAT], ATA settings applied: Standby:242
Jul 9 11:57:11 yggdrasil smartd[20542]: Device: /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408TYD [SAT], is SMART capable. Adding to "monitor" list.
Jul 9 11:57:11 yggdrasil smartd[20542]: Device: /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408TYD [SAT], state read from /var/lib/smartmontools/smartd.ST8000AS0002_1NA17Z-Z8408TYD.ata.state
…
Jul 9 11:57:11 yggdrasil smartd[20542]: Monitoring 8 ATA and 0 SCSI devices
…
Jul 9 11:57:22 yggdrasil smartd[20542]: Device: /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408DQT [SAT], state written to /var/lib/smartmontools/smartd.ST8...

[Nish Aravamudan (nacc)]

16.10 now has 6.5+svn4324-1.

[Nish Aravamudan (nacc)]

@Andrew, would you be willing to update this bug per the template at https://wiki.ubuntu.com/StableReleaseUpdates ?

[Mark Barsky (push-uni)]

Hello. Can someone confirm that 6.5+svn4324-1 version of smartmontools from 16.04 actually fixes the described issue?

[Andrew (caeci11iusad1)]

Yes - I can confirm that the version posted above fixes the issue from 16.04. I used it for between 4 and 6 months without issue, but upgraded to 16.10 (a few months after release) and now 17.04 (pretty well immediately upon release) which have later versions included.

[Bryce Harrington (bryce)]

[Standard support has ended for Trusty]