addition of zfsutils-linux scrib every 2nd sunday

[fermulator]

FULL REF: https://github.com/zfsonlinux/zfs/issues/9858 (initially submitted there mistakenly)

==System information==
{{{
Type Version/Name
Distribution Name Ubuntu
Distribution Version 18.04.3 LTS
Linux Kernel 4.15.0-73-generic
Architecture x86_64
ZFS Version 0.7.5-1ubuntu16.6
SPL Version 0.7.5-1ubuntu2
Describe the problem you're observing
}}}

==When did this file get added into zfsonlinux ecosystem?==

{{{
# cat /etc/cron.d/zfsutils-linux
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Scrub the second Sunday of every month.
24 0 8-14 * * root [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ] && /usr/lib/zfs-linux/scrub
}}}

I've been a ZoL user for many years now, and have had my own cron setup tailored to distribute pool scrubs once per month, spread across the month to avoid system I/O overload on any one day of the month, like this:
{{{
# zfsmain scrub: 2:30AM 7th of every month
30 02 7 * * /sbin/zpool scrub zfsmain

# zstorage scrub: 2:30AM 5th of every month
30 02 5 * * /sbin/zpool scrub zstorage

# zmedia scrub: 1:00AM 14th of every month
00 01 14 * * /sbin/zpool scrub zmedia

# zstorage scrub: 2:30AM 21st of every month
30 02 21 * * /sbin/zpool scrub ztank
}}}

However suddenly I noticed in an adhoc check of zpool status that ALL my pools were scrubbed on Sunday January 12th 2020!
{{{
# zpool status | grep -i "scrub"
  scan: scrub repaired 0B in 0h13m with 0 errors on Sun Jan 12 00:37:31 2020
  scan: scrub repaired 0B in 11h24m with 0 errors on Tue Jan 14 12:24:33 2020 <-- (one of my own since the Jan12 mega scrub)
  scan: scrub repaired 0B in 0h45m with 0 errors on Sun Jan 12 01:09:40 2020
  scan: scrub repaired 0B in 7h10m with 0 errors on Sun Jan 12 07:34:11 2020
}}}
this is NOT what I had configured, so I went digging and found that zfsutil cron file :(
{{{
# cat /usr/lib/zfs-linux/scrub
#!/bin/sh -eu

# Scrub all healthy pools.
zpool list -H -o health,name 2>&1 | \
 awk 'BEGIN {FS="\t"} {if ($1 ~ /^ONLINE/) print $2}' | \
while read pool
do
 zpool scrub "$pool"
done

# cat /etc/cron.d/zfsutils-linux
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Scrub the second Sunday of every month.
24 0 8-14 * * root [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ] && /usr/lib/zfs-linux/scrub
}}}

This MAY be a desirable default for some system admins, but having it suddenly appear IMO is at the same time undesirable for many.

==Describe how to reproduce the problem==

* Having been a ZoL user sys admin for many years.
* Be a decent sys admin and know the basics, you've configured your own cron/schedule for pool scrubbing per guidelines and individual needs.
* Follow stable upgrade channels. (for me this is Ubuntu LTS 16.04, then 18.04)
* Suddenly after some upgrade version XX(?) your pools start scrubbing on the 2nd Sunday of every month without your having configured it or asked for that.

==Expectations==

Hoping we can:

 1. Confirm when and why this was rolled out to all systems by default (does the explanation make sense? is it really OK to do this? how was it communicated?)
 2. Ensure "how to disable" is documented and supported (i.e. if I just delete that cron file, will some future upgrade replace and re-enable it?)

----

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: zfsutils-linux 0.7.5-1ubuntu16.7
ProcVersionSignature: Ubuntu 4.15.0-73.82-generic 4.15.18
Uname: Linux 4.15.0-73-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.9-0ubuntu7.10
Architecture: amd64
Date: Sat Jan 18 13:41:29 2020
InstallationDate: Installed on 2015-06-28 (1664 days ago)
InstallationMedia: Ubuntu-Server 14.04.2 LTS "Trusty Tahr" - Release amd64 (20150218.1)
SourcePackage: zfs-linux
UpgradeStatus: Upgraded to bionic on 2019-05-10 (253 days ago)
modified.conffile..etc.sudoers.d.zfs: [inaccessible: [Errno 13] Permission denied: '/etc/sudoers.d/zfs']

[Richard Laager (rlaager)]

This was added a LONG time ago. The interesting question here is: if you previously deleted it, why did it come back? Had you deleted it though? It sounds like you weren’t aware of this file.

You might want to edit it in place, even just to comment out the job. That would force dpkg to give you a conffile merge prompt instead of being able to silently put it back.

[fermulator (fermulator)]

@rlaager ref for "long time ago"? (Looking for release notes and specific version)

I've been reasonably diligent in monitoring my pool scrub activity, I would "guess" that maybe in the last 2-3 months I may have lost track. But also I only upgraded from Ubuntu 16.04 LTS sometime late-2019 Fall (iirc) - couldn't find a way to precisely determine when. -- the below details suggest in Sept 2019.

So "LONG time ago", certainly possible, but many users might not pick this up until an upgrade from 16.04 LTS to 18.04 LTS.

Here's the history of the package. (only have so much detail due to log rotation)

```
/var/log/apt$ for file in $(ls -1 | grep history.log | grep gz | sort -V); do echo "==="; echo; echo $file; ls -al $file | awk '{print $6, $7, $8}'; zgrep zfsutils-linux $file | tr ", " "\n" | grep -A1 zfsutils-linux; done
===

history.log.1.gz
Jan 1 00:00
===

history.log.2.gz
Nov 28 06:53
===

history.log.3.gz
Oct 31 06:39
===

history.log.4.gz
Sep 30 06:20
zfsutils-linux:amd64
(0.7.5-1ubuntu16.4
===

history.log.5.gz
Aug 30 06:57
===

history.log.6.gz
Jul 31 06:27
===

history.log.7.gz
Jun 30 2019
===

history.log.8.gz
May 31 2019
===

history.log.9.gz
Apr 30 2019
===

history.log.10.gz
Mar 30 2019
zfsutils-linux:amd64
(0.6.5.6-0ubuntu25
===

history.log.11.gz
Mar 1 2019
===

history.log.12.gz
Feb 1 2019
```

The above at least definitely proves that I was previously on 0.6.5.6 (likely while on Ubuntu 16.04) - where the zfsutils-linux cron scrub was NOT present to my knowledge -- then in Sep 2019 we jump to 0.7.5 (likely when I upgraded the server to Ubuntu 18.04).

To be clear, I've never deleted it, nor edited it, as only this week I uncovered the "problem".
I agree that deletion of the file may be ill-advised ... since the next time dpkg updates it would probably to re-add it?

The suggestion to edit the file and comment out the scrub command is likely safest, however then all future upgrades will complain about the conflict.

It would be preferable if there was a configuration to enable/disable it.

[Richard Laager (rlaager)]

zfs-linux (0.6.5.6-2) unstable; urgency=medium
...
  * Scrub all healthy pools monthly from Richard Laager

So Debian stretch, but not Ubuntu 16.04.

Deleting the file should be safe, as dpkg should retain that. It sounds like you never deleted it, as you didn’t have it before this upgrade. So it wasn’t an issue of the file coming back, just appearing for the first time. Deleting and editing conffiles is a standard thing in Debian systems.

These days, we may want to convert this to a system timer/service pair instead, which you could then disable/mask if you don’t want. Of course, the initial conversion will cause the same complaint you have here: something changed on upgrade and enabled a job you don’t want.

[Richard Laager (rlaager)]

[This comment was seemingly hidden?]

> The real question is, why is "[ $(date +\%w) -eq 0 ]" in there, when
cron can do day-of-week like:
>
> 24 0 8-14 * 0 root [ -x /usr/lib/zfs-linux/scrub ] && /usr/lib/zfs-
linux/scrub

This is because if you specify the "day of month" and the "day of week" fields, they are ORed, not ANDed. From crontab(5), "If both fields are restricted (i.e., aren't *), the command will be run when either field matches the current time."