for Seagate USB drive enclosures, SAT (e.g. smartmontools, hdparm) works on kernel 4.13 but not on 4.15

[Mike Bernson]

smartctl -d sat -i /dev/sda work on 16.04 kernel 4.13 but on 4.15.

Here is data from 16.04
root@server:~# uname -a
Linux server 4.13.0-45-generic #50~16.04.1-Ubuntu SMP Wed May 30 11:18:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@server:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial
root@server:~# smartctl -d sat -i /dev/sda
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.13.0-45-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: ST8000DM004-2CX188
Serial Number: ZCT07YN6
LU WWN Device Id: 5 000c50 0b1f2a4d2
Firmware Version: 0001
User Capacity: 8,001,563,222,016 bytes [8.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5425 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Jan 1 16:57:48 2019 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

root@server:~#

here is data from 18.04:root@mike-desktop:/tmp# uname -a
Linux mike-desktop 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@mike-desktop:/tmp# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic
root@mike-desktop:/tmp# smartctl -d sat -i /dev/sdd
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-36-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: scsi error unsupported field in scsi command

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

dmesg on 18.04 showing drive is connected /dev/sdd:
[6041017.984544] usb 3-2.4: new SuperSpeed USB device number 6 using xhci_hcd
[6041018.009417] usb 3-2.4: New USB device found, idVendor=0bc2, idProduct=331a
[6041018.009424] usb 3-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[6041018.009428] usb 3-2.4: Product: Expansion Desk
[6041018.009432] usb 3-2.4: Manufacturer: Seagate
[6041018.009435] usb 3-2.4: SerialNumber: NAAA7HWG
[6041018.015808] scsi host5: uas
[6041018.016527] scsi 5:0:0:0: Direct-Access Seagate Expansion Desk 0915 PQ: 0 ANSI: 6
[6041018.017645] sd 5:0:0:0: Attached scsi generic sg3 type 0
[6041030.357108] sd 5:0:0:0: [sdd] 15628053167 512-byte logical blocks: (8.00 TB/7.28 TiB)
[6041030.357112] sd 5:0:0:0: [sdd] 4096-byte physical blocks
[6041030.357265] sd 5:0:0:0: [sdd] Write Protect is off
[6041030.357268] sd 5:0:0:0: [sdd] Mode Sense: 53 00 00 08
[6041030.357562] sd 5:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[6041030.493932] sdd: sdd1 sdd9
[6041030.495310] sd 5:0:0:0: [sdd] Attached SCSI disk

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: smartmontools 6.5+svn4324-1
ProcVersionSignature: Ubuntu 4.15.0-36.39-generic 4.15.18
Uname: Linux 4.15.0-36-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
Date: Tue Jan 1 16:53:16 2019
InstallationDate: Installed on 2018-02-23 (312 days ago)
InstallationMedia: Ubuntu-GNOME 16.04.3 LTS "Xenial Xerus" - Release amd64 (20170801)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: smartmontools
UpgradeStatus: Upgraded to bionic on 2018-10-21 (72 days ago)

[Christian Franke (christian-franke)]

> Read Device Identity failed: scsi error unsupported field in scsi command
> ...
> [6041018.009417] usb 3-2.4: New USB device found, idVendor=0bc2, idProduct=331a
> ...
> [6041018.009432] usb 3-2.4: Manufacturer: Seagate
> ...
> [6041018.015808] scsi host5: uas

This is typical for Seagate USB devices on recent Linux kernels if UAS mode is enabled.

See info and related tickets about UAS on Linux here:
https://www.smartmontools.org/wiki/USB

[Mike Bernson (mike-mlb)]

It look like there was a patch added to fix this

https://github.com/torvalds/linux/commit/7fee72d5e8f1e7b8d8212e28291b1a0243ecf2f1

 /* All Seagate disk enclosures have broken ATA pass-through support */
 if (le16_to_cpu(udev->descriptor.idVendor) == 0x0bc2)
  flags |= US_FL_NO_ATA_1X;

Since the id is 0bc2 this fix should fix the problem do no known if in ubuntu 4.15.0-36.

If not is there any way to turn off uas ?

from lsusb
Bus 003 Device 006: ID 0bc2:331a Seagate RSS LLC

[Christian Franke (christian-franke)]

> If not is there any way to turn off uas ?

Try kernel parameter:
  usb_storage.quirks=0bc2:331a:u disables UAS transfer mode,
  usb_storage.quirks=0bc2:331a: disables all quirks.

The latter might be risky as kernel developers disabled SAT ATA PASS-THROUGH in UAS mode for some good reason.

Both reportedly work according to these comments:
https://www.smartmontools.org/ticket/971#comment:12
https://www.smartmontools.org/ticket/1092#comment:3

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Since this is a reported regression when upgrading kernels, and the fix is identified in the kernel (thanks mike!) I'm reassigning this to the kernel package.

Kernel team - setting Confirmed as the required upstream patch is already identified. I hope that's acceptable.

[Nathan Stratton Treadway (nathanst)]

@Mike/Robie:

Actually the "uas: Always apply US_FL_NO_ATA_1X quirk to Seagate devices" patch to the kernel is (part of) the cause of the current smartmontool failure rather than a fix for it.

The underlying problem is that most Seagate drive enclosures do not properly handle SAT (= "ATA pass-through") when in UAS mode, and so the kernel now (after that patch) disables SAT for all Seagate enclosures when UAS is in use.

That change allows those enclosures to work okay in UAS mode for normal data access (with the associated increase in performance) ... but since Smartmontools and other utilities (e.g. "hdparm") require SAT to work, the change breaks their functionality for those devices.

The current workaround is to use the kernel quirks to disable UAS for a particular device, so that the kernel falls back to the old usb-storage mode instead; this lets SAT work again (and thus Smartmontools, etc. function properly), but only at the expense of losing UAS performance for normal data access.

So in the long run it would be ideal for the kernel to have some way to more dynamically deal with these sorts of buggy devices (i.e. switching modes temporarily to allow SAT to work for special operations such as Smartmontools but use UAT most of the time)... but I haven't run across any discussion about such a patch anywhere, so it seems like people with these devices will be stuck with having to choose between SAT or UAS for the time being.

Anyway, there's now a Smartmontools Wiki page devoted to this specific topic: https://www.smartmontools.org/wiki/SAT-with-UAS-Linux .

[James Kingdon (james-kingdon)]

I got hit by this after a power outage cycled my file servers and presumably activated a previous automatic kernel update (the servers normally stay up for months at a time).

The kernel that is currently running is:

Linux oban8 4.14.111-odroidxu4 #2 SMP PREEMPT Wed May 8 17:30:01 CEST 2019 armv7l armv7l armv7l GNU/Linux

The temporary work-around documented on the Smartmontools Wiki worked for me (thanks for the link!), now I have to figure out how to make the change persistent on these ARM based systems :(

[Nathan Stratton Treadway (nathanst)]

On Thu, Aug 08, 2019 at 15:27:20 -0000, James Kingdon wrote:
> I got hit by this after a power outage cycled my file servers and
> presumably activated a previous automatic kernel update (the servers
> normally stay up for months at a time).
>
> The kernel that is currently running is:
>
> Linux oban8 4.14.111-odroidxu4 #2 SMP PREEMPT Wed May 8 17:30:01 CEST
> 2019 armv7l armv7l armv7l GNU/Linux

Interesting. Do you know what kernel version was running beforehand?

Also, what are the USB ids of the drive enclosure device(s) that were
affected? (Were they in fact Seagate drives?)

>
> The temporary work-around documented on the Smartmontools Wiki worked
> for me (thanks for the link!), now I have to figure out how to make the

Cool.

> change persistent on these ARM based systems :(

I don't use Odroid myself but did run across mentions of it in my
web research on this topic...

Does the "UASP" item in
  https://wiki.odroid.com/odroid-xu4/os_images/linux/ubuntu_4.14/20181203-minimal#known_issues_and_tips
point you in the right direction?

(After reading that paragraph, I'm curious to know if the problem you
hit was actually the NO_ATA_1X flag getting set on your device when
using the UAS driver [as originally covered in this bug], or a more
general failure of the UAS driver on ARM for your devices. That is,
did you come to this web page because Smartmontools stopped working, or
because your disks stopped working?

[In either case, the :u quirks would work around the problem; the
question is just which problem you are working around...])

      Nathan

[James Kingdon (james-kingdon)]

Hi Nathan, thanks for the reply.

I'm not sure what kernel was there before. I was expecting to see traces in /boot but the only references are to 4.14.133.

Yes, the drives are Seagate 4TB ones, ID 0bc2:331a

Thanks for the link to the known issues. I was looking at boot.ini so it seems I was heading in the right direction. I'm not sure if the copy I can see in /boot is active during the boot process or not, or if I need some additional steps to make that happen. I'm sure I can dig that out with the help of Google.

I came to this page in a typically long and circuitous fashion after I noticed that the external drivers weren't spinning down anymore and discovering that hdparm wasn't working. Basic file IO was still ok.

[Nathan Stratton Treadway (nathanst)]

On Thu, Aug 08, 2019 at 18:05:55 -0000, James Kingdon wrote:
> I'm not sure what kernel was there before. I was expecting to see traces
> in /boot but the only references are to 4.14.133.

(Weird. I'm not familiar with the kernels for odroid, but off hand I'd
be suprised if the Seagate-blacklisting behavior would have changed
between this kernel version and previous ones within the Ubuntu 18.04
release.... I wonder if the full explanation is even more complex, e.g.
that actually hdparm was failing in the previous kernel versions too,
but the power-saving settings on the drives in the enclosures was
retained from before... until you lost power and the drives did a fresh
restart -- or something.)

> Yes, the drives are Seagate 4TB ones, ID 0bc2:331a
[...]
> I came to this page in a typically long and circuitous fashion after I
> noticed that the external drivers weren't spinning down anymore and
> discovering that hdparm wasn't working. Basic file IO was still ok.

Okay, yes, definitely the NO_ATA_1X problem.

Ah, were you using the "hdparm -B" or "-S" options (or the corresponding
apm=/spindown_time= settings in the Ubuntu package's hdparm.conf file)
to set the spin-down behavior? That's another use case affected by the
kernel's current NO_ATA_1X situation....

       Nathan

[James Kingdon (james-kingdon)]

Yes, hdparm -S. Not sure if it's relevant, but the distro is 16.04.6 LTS (Xenial Xerus) in this case.

[Nathan Stratton Treadway (nathanst)]

On Thu, Aug 08, 2019 at 18:57:24 -0000, James Kingdon wrote:
> Yes, hdparm -S.

Okay, thanks.

> Not sure if it's relevant, but the distro is 16.04.6 LTS
> (Xenial Xerus) in this case.

Okay, yeah, that makes a little more sense. Again, I don't know the
timeline on these specific kernels, but in the kernel.org kernel
branches the Seagate NO_ATA_1X patch seems to have been applied in
Nov/Dec 2017... so I would guess that it was included-from-the-start in
the Ubuntu 18.04 kernels, but it would make sense for it to have been
newly-added over the lifetime of the 16.04 distribution.

(Out of curiousity, what's the path to the package repository given in
your /etc/apt/sources.list file? The wiki.odroid.com pages have links
for downloading the "install Ubuntu OS" image, but I have't found any
information about where the associated package repository is located...)

       Nathan

[James Kingdon (james-kingdon)]

source.list is using http://ports.ubuntu.com/

[Nathan Stratton Treadway (nathanst)]

On Thu, Aug 08, 2019 at 21:59:50 -0000, James Kingdon wrote:
> source.list is using http://ports.ubuntu.com/

Okay, interesting. So I guess most of your install is the armhf
architecture packages from ports.ubuntu.com. However, the the kernel
you mentioned (4.14.111-odroidxu4) isn't from there; I guess there must
be another source.list line pointing to http://deb.odroid.in/5422-s/ or
something?

(If that is indeed the origin of your kernel packages, it does look like
the versons that have been published there do span across the kernel
versions were the Seagate NO_ATA_1X patch was introduced....)

     Nathan

[James Kingdon (james-kingdon)]

Hmm, you've got me curious now. It's been a while since I really looked at the ARM boards. sources.list.d contains armbian.list which has a single entry:

deb http://apt.armbian.com xenial main xenial-utils xenial-desktop

That seems like a reasonable candidate? I don't recall if I used an original distro from Odroid or (more likely) went straight for a release from armbian - they were the my favourite when I was building an ARM farm a couple of years ago.

References to odroid in /etc/apt seem to be limited to apt.conf.d/01autoremove-kernels. That might answer your earlier question about what the previous kernel level was as it mentions 4.14.111.

[Nathan Stratton Treadway (nathanst)]

On Thu, Aug 08, 2019 at 23:50:14 -0000, James Kingdon wrote:
> Hmm, you've got me curious now. It's been a while since I really looked
> at the ARM boards. sources.list.d contains armbian.list which has a
> single entry:
>
> deb http://apt.armbian.com xenial main xenial-utils xenial-desktop
>
> That seems like a reasonable candidate? I don't recall if I used an

Well... I off hand don't see 4.14.111-odroidxu4 on the apt.armbian.com
site at the moment (only 5.x packages), but that doesn't prove much
about what was there in the past...

(But if you are indeed using Armbian ["lsb_release -a" should tell you],
then the link to the wiki.odroid.com Ubuntu release notes won't be of
any help... seems like /boot/ArmbianEnv.txt may be what you are looking
for instead.)

> References to odroid in /etc/apt seem to be limited to apt.conf.d
> /01autoremove-kernels. That might answer your earlier question about
> what the previous kernel level was as it mentions 4.14.111.

(In your previous email you said the running kernel version was
"4.14.111-odroidxu4", so I think it's mentioned in 01autoremove-kernels
simply because it's the currently-installed package version.... but we
probably don't need to try any harder tracking down the history of the
Armbian kernel releases here in an Ubuntu bug report... :) )

       Nathan

[James Kingdon (james-kingdon)]

"but we
probably don't need to try any harder tracking down the history of the
Armbian kernel releases here in an Ubuntu bug report..."

Agreed, so the following is just out of interest...
re kernel versions, sorry for the confusion, it looks like the reboot I did today moved me up another kernel release without me realising, so I was on .111 at the beginning of the day and .113 at the end. No idea what might have been there back when things were working, but as you say, it's not so important.

lsb_release -a doesn't mention armbian:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

But the login message does:

  ___ _ _ _ __ ___ _ _ _
 / _ \ __| |_ __ ___ (_) __| | \ \/ / | | | || |
| | | |/ _` | '__/ _ \| |/ _` | \ /| | | | || |_
| |_| | (_| | | | (_) | | (_| | / \| |_| |__ _|
 \___/ \__,_|_| \___/|_|\__,_| /_/\_\\___/ |_|

Welcome to Ubuntu Xenial with Armbian Linux 4.14.133-odroidxu4

And I should probably stop spamming this bug report with irrelevant noise now. It seems like the problem is well understood, it's a shame there isn't a simpler way of choosing between the uas and usb-storage drivers, but at least it can be done.

Many thanks for all the help.

[Mikko Rantalainen (mira)]

For more information about this issue, see https://askubuntu.com/a/1106020/50254

TL;DR: Linux kernel disables 12 and 16 bit SATA Commands over UAS when used with Seagate HDD enclosures. This usb_storage quirk is "t" instead of "u" which forces the device into USB BOT mode. The only way to make S.M.A.R.T. to work with USB HDD enclosure is to use UAS without limitations - that is, no quirk of any kind.

Note that full UAS support has been disabled on all Seagate enclosures by default because Seagate had such a poor track record with multiple products. The quirk "t" is enough to avoid the issue and it will not cause equal performance slowdown as the "u" quirk. Note that if you disable all quirks with known-bad Seagate enclosure the S.M.A.R.T. will work but you risk random data corruption if you write any data on the disk. I guess remounting the disk read-only, disabling the "t" quirk, doing S.M.A.R.T. query, re-enable the "t" quirk and remounting read-write could be considered a safe workaround if you really need the S.M.A.R.T. data.

More details about possible usb_storage.quirks:
https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt

[Mikko Rantalainen (mira)]

See also: https://www.smartmontools.org/wiki/SAT-with-UAS-Linux