Focal installer destroyed my datasets

[Theo Markettos]

I installed focal about a month about, using the desktop installer with the 'experimental' ZFS root option. I've been running ZFS on FreeBSD for about 10 years and figured I could handle any sharp edges. It's installed to a single 2TB SSD - no RAID, no L2ARC/ZIL, no dedup, nothing complicated. Machine is an Intel S2600CP2, 2x E5-2670 v1, 128GiB RAM.

The machine stayed up for a few weeks, until I had cause to physically move it and so shut it down. It turned out GRUB had installed onto the wrong disk which I'd removed in the intervening time. It wouldn't boot, so I booted the focal desktop installer from a USB stick to give me a shell to fix things. I didn't run the actual installer, I just booted to the installer desktop and then did Ctrl-Alt-F1, apt install openssh-server, and then logged in via SSH.

Then I followed some instructions about mounting the pool in a chroot (Step 3):
https://askubuntu.com/questions/826209/re-initialise-grub-for-non-bootable-uefi-zfs-16-04-installation
This took a couple of goes to get right as the first time I tried to mount on top of the installer (without zpool import -R) and many filesystems couldn't replace preexisting mounts, so I exported and re-imported. Anyway, I succeeded in mounting, and then doing update-grub. However, looking at the zpool history -il log while I was in the installer I see:
(full zpool history -il log attached)

2020-06-16.14:56:34 [txg:889337] open pool version 5000; software version unknown; uts ubuntu 5.4.0-26-generic #30-Ubuntu SMP Mon Apr 20 16:58:30 UTC 2020 x86_64 [on ubuntu]
2020-06-16.14:56:34 [txg:889339] import pool version 5000; software version unknown; uts ubuntu 5.4.0-26-generic #30-Ubuntu SMP Mon Apr 20 16:58:30 UTC 2020 x86_64 [on ubuntu]
2020-06-16.14:58:20 zpool export rpool bpool [user 0 (root) on ubuntu:linux]
2020-06-16.14:59:16 [txg:889372] open pool version 5000; software version unknown; uts ubuntu 5.4.0-26-generic #30-Ubuntu SMP Mon Apr 20 16:58:30 UTC 2020 x86_64 [on ubuntu]
2020-06-16.14:59:16 [txg:889374] import pool version 5000; software version unknown; uts ubuntu 5.4.0-26-generic #30-Ubuntu SMP Mon Apr 20 16:58:30 UTC 2020 x86_64 [on ubuntu]
2020-06-16.15:01:32 [txg:889402] destroy rpool/ROOT/ubuntu_rjlt3b/var/log@autozsys_1gwqtf (19222) [on ubuntu]
2020-06-16.15:01:32 ioctl destroy_snaps
    input:
        snaps:
            rpool/ROOT/ubuntu_rjlt3b/var/log@autozsys_1gwqtf
 [user 0 (root) on ubuntu:linux]

...lots of snapshots are deleted...

2020-06-16.15:05:55 [txg:889577] destroy rpool/USERDATA/root_tavly8@autozsys_1gwqtf (19722) [on ubuntu]
2020-06-16.15:06:00 ioctl destroy_snaps
    input:
        snaps:
            rpool/USERDATA/root_tavly8@autozsys_1gwqtf
2020-06-16.15:06:45 [txg:889588] destroy rpool/USERDATA/local/vm/freebsd-12.1 (3398) [on ubuntu]
2020-06-16.15:06:45 [txg:889590] destroy rpool/USERDATA/local/vm (735) [on ubuntu]
2020-06-16.15:06:46 [txg:889592] destroy rpool/USERDATA/local/ecad/altera (551) [on ubuntu]
2020-06-16.15:06:46 [txg:889594] destroy rpool/USERDATA/local/ecad/xilinx (3476) [on ubuntu]
2020-06-16.15:06:47 [txg:889596] destroy rpool/USERDATA/local/ecad (127) [on ubuntu]
2020-06-16.15:06:48 [txg:889598] destroy rpool/USERDATA/local/scratch/atm26 (2058) [on ubuntu]
2020-06-16.15:06:48 [txg:889600] destroy rpool/USERDATA/local/scratch (2447) [on ubuntu]
2020-06-16.15:06:49 [txg:889602] destroy rpool/USERDATA/local (1671) [on ubuntu]
2020-06-16.15:07:20 [txg:889668] destroy rpool/USERDATA/opt_tavly8/riscv (1878) [on ubuntu]
2020-06-16.15:07:21 [txg:889670] destroy rpool/USERDATA/opt_tavly8/brave.com (1864) [on ubuntu]
2020-06-16.15:07:23 [txg:889672] destroy rpool/USERDATA/opt_tavly8/riscv-llvm (4788) [on ubuntu]
2020-06-16.15:07:24 [txg:889674] destroy rpool/USERDATA/opt_tavly8/riscv-freebsd (5147) [on ubuntu]
2020-06-16.15:07:25 [txg:889676] destroy rpool/USERDATA/opt_tavly8 (3313) [on ubuntu]

Here something that wasn't the command line - and so I assume is zsys - deleted all the datasets I made in rpool/USERDATA. This was about 1TiB of data.

Needless to say, nothing should be destroying user datasets.

root@mesozoic:/# lsb_release -rd
Description: Ubuntu 20.04 LTS
Release: 20.04
root@mesozoic:/# apt-cache policy zsys
zsys:
  Installed: 0.4.5
  Candidate: 0.4.5
  Version table:
 *** 0.4.5 500
        500 http://ubuntu.mirrors.uk2.net/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status
root@mesozoic:/# dpkg --list | grep zfs
ii libzfs2linux 0.8.3-1ubuntu12 amd64 OpenZFS filesystem library for Linux
ii zfs-initramfs 0.8.3-1ubuntu12 amd64 OpenZFS root filesystem capabilities for Linux - initramfs
ii zfs-zed 0.8.3-1ubuntu12 amd64 OpenZFS Event Daemon
ii zfsutils-linux 0.8.3-1ubuntu12 amd64 command-line tools to manage OpenZFS filesystems

[Theo Markettos (caliston)]

Also to note, not all the datasets I created in rpool/USERDATA were destroyed:

oot@mesozoic:/# zfs list | grep USERDATA
rpool/USERDATA 338G 1.41T 96K /
rpool/USERDATA/atm26_tavly8 32.6G 1.41T 29.9G /home/atm26
rpool/USERDATA/root_tavly8 920K 1.41T 592K /root
rpool/USERDATA/ssd250 305G 1.41T 88K /local/ssd250
rpool/USERDATA/ssd250/build 55.3G 1.41T 112K /local/ssd250/build
rpool/USERDATA/ssd250/build/lineageos 53.1G 1.41T 52.9G /local/ssd250/build/lineageos
rpool/USERDATA/ssd250/build/riscos 2.17G 1.41T 2.17G /local/ssd250/build/riscos
rpool/USERDATA/ssd250/docker 746M 1.41T 746M /local/ssd250/docker
rpool/USERDATA/ssd250/dvd 458M 1.41T 458M /local/ssd250/dvd
rpool/USERDATA/ssd250/vm 249G 1.41T 88K /local/ssd250/vm
rpool/USERDATA/ssd250/vm/ecadlabs 152K 1.41T 88K /local/ssd250/vm/ecadlabs
rpool/USERDATA/ssd250/vm/jenkins 240G 1.41T 240G /local/ssd250/vm/jenkins
rpool/USERDATA/ssd250/vm/uvtool 8.74G 1.41T 8.74G /local/ssd250new/vm/uvtool

[Didier Roche-Tolomelli (didrocks)]

This was fixed in zsys 0.4.6.