[Wily] wakes up green drives without a valid reason

[Ralf Mardorf]

Hi,

a minimalist Wily server install, around 4.5 GiB small does cause endless spin downs and spin ups, so that green drives get damaged. I maintain an around 37.5 GiB large Arch Linux install, quasi using the same software used by the Wily install and in addition a little bit more software, which isn't used by the Wily install. For Arch Linux I don't experience this issue. I expect the same for an Ubuntu install.

I'm already hunting the culprit, see
https://lists.ubuntu.com/archives/ubuntu-studio-devel/2015-August/006570.html and
https://lists.ubuntu.com/archives/ubuntu-studio-devel/2015-August/006571.html,
but run out of ideas.

The claim the install is 2 GiB small is a typo, the backup archive is of that size.

However, assumed Ubuntu won't contribute to http://andrewmcconnell.photoshelter.com/gallery/G0000oLuiBLHIsmM this "feature" should be removed from the defaults.

$ lsb_release -rd
Description: Ubuntu Wily Werewolf (development branch)
Release: 15.10

Regards,
Ralf

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1484497/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1484497

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Ralf Mardorf (ralf-mardorf-j)]

Please mention the log files you need and I'll post them. For good
reasons I removed apport.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.2 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.2-rc7-unstable/

[Ralf Mardorf (ralf-mardorf-j)]

Thank you,

unfortunately it didn't fix the issue.

[root@moonstudio weremouse]# t=10800;y=$(smartctl -A /dev/sdc|grep Lo|awk '{print $NF}');sleep $t;x=$(smartctl -A /dev/sdc|grep Lo|awk '{print $NF}');echo $(uname -r) $x-$y=$((x-y)) spins in $(($t/60/60)) hours
4.2.0-040200rc7-generic 15287-15280=7 spins in 3 hours

This issue never appeared related to kernel configs or kernel releases. I suspect the culprit gets executed at startup.

[Ralf Mardorf (ralf-mardorf-j)]

To rule out the kernel being the culprit, I build a kernel I'm using with Arch Linux, were green drives stay asleep. IMO it's not caused by the package linux.
There are no differences for the config, that are relevant for the issue.
It's unlikely that there were any additional patches used for the Arch Linux kernel, that are related to the issue.

The spin down/spin up test:

[root@moonstudio weremouse]# t=10800;y=$(smartctl -A /dev/sdc|grep Lo|awk '{print $NF}');sleep $t;x=$(smartctl -A /dev/sdc|grep Lo|awk '{print $NF}');printf "\n$(uname -rm)$(lsb_release -d|cut -f2 -d:|cut -f1 -d"(")\n$x-$y=$((x-y)) spins in $(($t/60/60)) hours\n"

3.10.61-rt65-1-moonstudio x86_64 Ubuntu Wily Werewolf
15342-15333=9 spins in 3 hours

Arch Linux only makes one spin, when smartctl wakes up the drive after 3 hours.

That's how I build the Ubuntu kernel:

[root@moonstudio src]# export CONCURRENCY_LEVEL=2;wget
https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.61.tar.gz
https://www.kernel.org/pub/linux/kernel/projects/rt/3.10/older/patch-3.10.61-rt65.patch.gz
&& tar zxf linux-3.10.61.tar.gz
&& mv linux-3.10.61 linux-3.10.61-rt65
&& cd linux-3.10.61-rt65 && gzip -dc ../patch-3.10.61-rt65.patch.gz | patch -p1
&& cp ../config-3.10.61-rt65-1-rt-lts.01.edit .config
&& make oldconfig && make menuconfig && make oldconfig
&& date
&& make-kpkg clean && make-kpkg --initrd kernel-image kernel-headers && make-kpkg clean
&& date

Most likely the Arch kernel just had two additional patches:

change-default-console-loglevel.patch
fix-race-in-PRT-wait-for-completion-simple-wait-code_Nvidia-RT.patch

Both kernels are configured with CONFIG_X86_64=y and CONFIG_X86=y.
That's the diff of the Arch and Ubuntu config:

$ diff config-3.10.61-rt65-1-rt-lts_x86_64.Arch_Linux_original /boot/config-3.10.61-rt65-1-moonstudio
3c3
< # Linux/x86 3.10.61 Kernel Configuration
---
> # Linux/x86_64 3.10.61 Kernel Configuration
52c52
< CONFIG_LOCALVERSION="-1-rt-lts"
---
> CONFIG_LOCALVERSION="-1-moonstudio"
162c162
< CONFIG_CHECKPOINT_RESTORE=y
---
> # CONFIG_CHECKPOINT_RESTORE is not set
321d320
< CONFIG_ASN1=m
1734c1733
< CONFIG_SCSI_ADVANSYS=m
---
> # CONFIG_SCSI_ADVANSYS is not set
1738,1740c1737,1739
< CONFIG_MEGARAID_MAILBOX=m
< CONFIG_MEGARAID_LEGACY=m
< CONFIG_MEGARAID_SAS=m
---
> # CONFIG_MEGARAID_MAILBOX is not set
> # CONFIG_MEGARAID_LEGACY is not set
> # CONFIG_MEGARAID_SAS is not set
2179c2178
< CONFIG_PCH_GBE=m
---
> # CONFIG_PCH_GBE is not set
4116,4117c4115
< CONFIG_DRM_I915=m
< CONFIG_DRM_I915_KMS=y
---
> # CONFIG_DRM_I915 is not set
5186c5184
< CONFIG_R8187SE=m
---
> # CONFIG_R8187SE is not set
5192,5193c5190,5191
< CONFIG_RTL8192E=m
< CONFIG_R8712U=m
---
> # CONFIG_RTL8192E is not set
> # CONFIG_R8712U is not set
5207,5208c5205,5206
< CONFIG_WLAGS49_H2=m
< CONFIG_WLAGS49_H25=m
---
> # CONFIG_WLAGS49_H2 is not set
> # CONFIG_WLAGS49_H25 is not set
6040,6042c6038
< CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE=m
< CONFIG_PUBLIC_KEY_ALGO_RSA=m
< CONFIG_X509_CERTIFICATE_PARSER=m
---
> # CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE is not set
6125d6120
< CONFIG_CLZ_TAB=y
6128d6122
< CONFIG_MPILIB=m

[Ralf Mardorf (ralf-mardorf-j)]

It's questionable if it should be considered a bug or not. The culprit is smartd. Ironically I used the package smartmontools 6.3+svn4002-2 to get smartctl, for testing if something wakes up a green drive. Despite the policy to autostart everything by default, that can be autostarted for better or worse, it's to consider if the default configuration, to poll every 30 minutes, is a wise decision. By an EU Regulation all external hard disk drives are forced to spin down and go to sleep after a while. I disabled smartd.service, any unneeded service is unwanted on my machine.

After disabling it, nothing does wake up a green drive anymore.

[root@moonstudio weremouse]# echo;date;t=10800;y=$(smartctl -A /dev/sdc|grep Lo|awk '{print $NF}');sleep $t;x=$(smartctl -A /dev/sdc|grep Lo|awk '{print $NF}');printf "\n$(uname -rm)$(lsb_release -d|cut -f2 -d:|cut -f1 -d"(")\n$x-$y=$((x-y)) spins in $(($t/60/60)) hours\n\n";date

Mon Aug 24 15:00:50 CEST 2015

3.10.61-rt65-1-moonstudio x86_64 Ubuntu Wily Werewolf
15377-15376=1 spins in 3 hours

Mon Aug 24 18:00:56 CEST 2015

The only spin is caused by smartctl, when it was executed on demand, to finish the test.

How I noticed that smartd is the culprit:
https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2015-August/015782.html

A note regarding smartd:
https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2015-August/015784.html

[Christian Ehrhardt  (paelzer)]

Hi,
I'm clearing out old open issues - I wanted to thank you for your debugging on this.
I think you are right and I also agree to be unsure if that should be considered a bug or not.
It is essentially a consequence of the device monitoring which people install to check if it wakes up - that makes sense.

For now I'll mark it "opinion" for now to keep it around if one gets by the issue again with another opinion to discuss.