zfs fails reverting to a previous snapshot on reboot when selected on grub

Bug #1946808 reported by BertN45
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
zfs-linux (Ubuntu)
Fix Released
Critical
Dimitri John Ledkov
Impish
Fix Released
Critical
Dimitri John Ledkov

Bug Description

[Impact]

 * zfs fails reverting to a previous snapshot on reboot when selected on grub

 * A miss-merge dropped initramfs hook changes, which result in failing to generate and use a new zfs uid.

 * Thus revert code to how it was before in https://launchpad.net/ubuntu/+source/zfs-linux/2.0.2-1ubuntu3

[Test Plan]

 * Create snapshot with $ zsysctl save test-sru --system

 * Reboot and attempt to boot into test-sru snapshot

[Where problems could occur]

 * The change is to initramfs hook only, thus initrd generation and boot paths are affected. Only latest initrd is rebuilt, rather than old ones, meaning booting initrds of older kernel abis will not be resolved.

[Other Info]

 * Original bug report

After creating a snapshot with: zsysctl save 211012-linux13-19 -s
the reboot fails as shown on the screenshot, the other screenshot shows the result of the snapshot.

ProblemType: Bug
DistroRelease: Ubuntu 21.10
Package: zsys 0.5.8
ProcVersionSignature: Ubuntu 5.13.0-19.19-generic 5.13.14
Uname: Linux 5.13.0-19-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu70
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: XFCE
Date: Tue Oct 12 19:11:43 2021
InstallationDate: Installed on 2021-10-12 (0 days ago)
InstallationMedia: Xubuntu 21.10 "Impish Indri" - Release amd64 (20211012)
Mounts: Error: [Errno 40] Too many levels of symbolic links: '/proc/mounts'
ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu_zgtuq6@/vmlinuz-5.13.0-19-generic root=ZFS=rpool/ROOT/ubuntu_zgtuq6 ro quiet splash
RelatedPackageVersions:
 zfs-initramfs 2.0.6-1ubuntu2
 zfsutils-linux 2.0.6-1ubuntu2
SourcePackage: zsys
SystemdFailedUnits:

UpgradeStatus: No upgrade log present (probably fresh install)
ZFSImportedPools:
 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
 bpool 768M 79.2M 689M - - 0% 10% 1.00x ONLINE -
 rpool 14G 3.33G 10.7G - - 1% 23% 1.00x ONLINE -
ZFSListcache-bpool:
 bpool /boot off on on off on off on off - none - - - - - - - -
 bpool/BOOT none off on on off on off on off - none - - - - - - - -
 bpool/BOOT/ubuntu_zgtuq6 /boot on on on off on off on off - none - - - - - - - -
ZSYSJournal:
 -- Journal begins at Tue 2021-10-12 18:10:37 AST, ends at Tue 2021-10-12 19:11:52 AST. --
 -- No entries --
modified.conffile..etc.apt.apt.conf.d.90_zsys_system_autosnapshot: [deleted]

Revision history for this message
BertN45 (lammert-nijhof) wrote :
Revision history for this message
BertN45 (lammert-nijhof) wrote :

Another screenshot with zfs list - t snapshot and some info in Conky

Revision history for this message
Didier Roche-Tolomelli (didrocks) wrote (last edit ):

The issue is in zfs-linux, where the merge from debian http://launchpadlibrarian.net/535966758/zfs-linux_2.0.2-1ubuntu5_2.0.3-8ubuntu1.diff.gz once again reverted some of the fixes and rolled back the patch to an earlier version. The fix was already reverted erronously in hirsute during the debian merge and we reintroduce the fix in https://launchpad.net/ubuntu/+source/zfs-linux/2.0.2-1ubuntu3.

Colin, do you mind having a look and reintroducing the patch as a 0-days SRU (the first time we introduced it was in https://launchpad.net/ubuntu/+source/zfs-linux/0.8.4-1ubuntu14)?

Can you check that you haven’t reverted by error other part of the patch and fix this one?
As this is happening consecutively in 2 releases where the debian merge doesn’t seem to start from the latest version in ubuntu but reintroduce an older version of the patch, can you have a look at the local setup issue you may have when doing the merges?

affects: zsys (Ubuntu) → zfs-linux (Ubuntu)
Changed in zfs-linux (Ubuntu):
assignee: nobody → Colin Ian King (colin-king)
summary: - zsys fail during reboot
+ zsys fail reverting to a previous snapshot on reboot
summary: - zsys fail reverting to a previous snapshot on reboot
+ zfs fails reverting to a previous snapshot on reboot when selected on
+ grub
Changed in zfs-linux (Ubuntu):
importance: Undecided → High
status: New → Incomplete
status: Incomplete → Triaged
Revision history for this message
Colin Ian King (colin-king) wrote : Re: [Bug 1946808] Re: zsys fail during reboot

Hi Didier,

Please can you pass this on to xnox as he's now handling all ZFS issues.
I'm leaving Canonical at the end of the week and won't be able to handle
any more SRUs.

Colin

On 13/10/2021 06:56, Didier Roche wrote:
> The issue is in zfs-linux, where the merge from debian
> http://launchpadlibrarian.net/535966758/zfs-
> linux_2.0.2-1ubuntu5_2.0.3-8ubuntu1.diff.gz once again reverted some of
> the fixes and rolled back the patch to an earlier version. The fix was
> already reverted erronously in hirsute during the debian merge and we
> reintroduce the fix in https://launchpad.net/ubuntu/+source/zfs-
> linux/2.0.2-1ubuntu3.
>
> Colin, do you mind having a look and reintroducing the patch as a 0-days
> SRU (the first time we introduced it was in
> https://launchpad.net/ubuntu/+source/zfs-linux/0.8.4-1ubuntu14)?
>
> Can you check that you haven’t reverted by error other part of the patch and fix this one?
> As this is happening consecutively in 2 releases where the debian merge doesn’t seem to start from the latest version in ubuntu but reintroduce an older version of the patch, can you have a look at the local setup issue you may have when doing the merges?
>
> ** Package changed: zsys (Ubuntu) => zfs-linux (Ubuntu)
>
> ** Changed in: zfs-linux (Ubuntu)
> Assignee: (unassigned) => Colin Ian King (colin-king)
>
> ** Summary changed:
>
> - zsys fail during reboot
> + zsys fail reverting to a previous snapshot on reboot
>
> ** Summary changed:
>
> - zsys fail reverting to a previous snapshot on reboot
> + zfs fails reverting to a previous snapshot on reboot when selected on grub
>
> ** Changed in: zfs-linux (Ubuntu)
> Importance: Undecided => High
>
> ** Changed in: zfs-linux (Ubuntu)
> Status: New => Incomplete
>
> ** Changed in: zfs-linux (Ubuntu)
> Status: Incomplete => Triaged
>

Changed in zfs-linux (Ubuntu):
assignee: Colin Ian King (colin-king) → Dimitri John Ledkov (xnox)
importance: High → Critical
description: updated
Revision history for this message
Dimitri John Ledkov (xnox) wrote (last edit ):

Also I wonder if:

grep -a -m10 -E "\*" /dev/urandom 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6

can be implemented as:

cut -c-6 /proc/sys/kernel/random/uuid

Revision history for this message
Dimitri John Ledkov (xnox) wrote :
description: updated
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello BertN45, or anyone else affected,

Accepted zfs-linux into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/2.0.6-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in zfs-linux (Ubuntu Impish):
status: Triaged → Fix Committed
Revision history for this message
BertN45 (lammert-nijhof) wrote :

The change worked on my PC. The difference has been tested with limited changes to my userdata, but I try tomorrow with the next set of updates. The current snapshots are:
bertadmin@VM-Xubuntu-2110:~$ zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
bpool/BOOT/ubuntu_q3cgtj@211015-proposed 56K - 78.6M -
rpool/ROOT/ubuntu_q3cgtj@211015-proposed 54.0M - 3.51G -
rpool/ROOT/ubuntu_q3cgtj/srv@211015-proposed 0B - 96K -
rpool/ROOT/ubuntu_q3cgtj/usr@211015-proposed 0B - 96K -
rpool/ROOT/ubuntu_q3cgtj/usr/local@211015-proposed 0B - 144K -
rpool/ROOT/ubuntu_q3cgtj/var@211015-proposed 0B - 96K -
rpool/ROOT/ubuntu_q3cgtj/var/games@211015-proposed 0B - 96K -
rpool/ROOT/ubuntu_q3cgtj/var/lib@211015-proposed 300K - 33.6M -
rpool/ROOT/ubuntu_q3cgtj/var/lib/AccountsService@211015-proposed 56K - 100K -
rpool/ROOT/ubuntu_q3cgtj/var/lib/NetworkManager@211015-proposed 104K - 140K -
rpool/ROOT/ubuntu_q3cgtj/var/lib/apt@211015-proposed 0B - 79.6M -
rpool/ROOT/ubuntu_q3cgtj/var/lib/dpkg@211015-proposed 256K - 37.9M -
rpool/ROOT/ubuntu_q3cgtj/var/log@211015-proposed 416K - 71.3M -
rpool/ROOT/ubuntu_q3cgtj/var/mail@211015-proposed 0B - 96K -
rpool/ROOT/ubuntu_q3cgtj/var/snap@211015-proposed 0B - 96K -
rpool/ROOT/ubuntu_q3cgtj/var/spool@211015-proposed 72K - 112K -
rpool/ROOT/ubuntu_q3cgtj/var/www@211015-proposed 0B - 96K -
rpool/USERDATA/bertadmin_g64ltk@211015-proposed 1.91M - 177M -
rpool/USERDATA/bertadmin_g64ltk@autozsys_6g35l0 128K - 176M -
rpool/USERDATA/root_g64ltk@211015-proposed 8K - 812K -
bertadmin@VM-Xubuntu-2110:~$

That autosys snapshot has been created automatically the first time I did run the next script to create a snapshot:

zsysctl save $1 -s
zfs list -t snapshot | grep rpool
zfs list -t snapshot | grep bpool

According to the Internet the next commands would stop all autosys snapshots

systemctl --user stop zsys-user-savestate.timer
systemctl --user disable zsys-user-savestate.timer
sudo mv /etc/apt/apt.conf.d/90_zsys_system_autosnapshot /etc/apt/apt.conf.d /90_zsys_system_autosnapshot_disabled

Did I make an error or is it another bug?

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 2.0.6-1ubuntu3

---------------
zfs-linux (2.0.6-1ubuntu3) jammy; urgency=medium

  * No-change rebuild to add python3.10.

 -- Matthias Klose <email address hidden> Sat, 16 Oct 2021 22:01:03 +0200

Changed in zfs-linux (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
BertN45 (lammert-nijhof) wrote :

I did run an update of only one file and I did install another application. Rebooting and reverting back to the last snapshot did work as expected. The application has been removed and I could rerun the update again.

According to the Internet the next commands would stop all autozsys snapshots

systemctl --user stop zsys-user-savestate.timer
systemctl --user disable zsys-user-savestate.timer
sudo mv /etc/apt/apt.conf.d/90_zsys_system_autosnapshot /etc/apt/apt.conf.d /90_zsys_system_autosnapshot_disabled

but I still have collected a lot of autozsys snapshots, see next list

rpool/USERDATA/bertadmin_9ykbc6@211015-proposed 2.08M - 177M -
rpool/USERDATA/bertadmin_9ykbc6@autozsys_h1yy2t 92K - 176M -
rpool/USERDATA/bertadmin_9ykbc6@211017-proposed 84K - 176M -
rpool/USERDATA/bertadmin_g64ltk@autozsys_6g35l0 456K - 176M -
rpool/USERDATA/bertadmin_g64ltk@autozsys_501rmb 520K - 176M -
rpool/USERDATA/bertadmin_g64ltk@autozsys_jr1n7t 488K - 176M -
rpool/USERDATA/bertadmin_g64ltk@autozsys_m29pbs 480K - 176M -
rpool/USERDATA/bertadmin_g64ltk@autozsys_4h79sx 356K - 175M -
rpool/USERDATA/root_9ykbc6@211015-proposed 72K - 812K -
rpool/USERDATA/root_9ykbc6@211017-proposed 0B - 812K -

Did I make an error or should I produce another bug-report?

Mathew Hodson (mhodson)
tags: added: verification-needed-impish
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Installed encrypted zsys impish system, enabled proposed, and applied all updates.

$ dpkg-query -W zfs-initramfs
zfs-initramfs 2.0.6-1ubuntu2.1

Created zsys snapshot and booted into it successfully.

Downgraded all zfs packages back to 2.0.6-1ubuntu2, to check that I can reproduce original failure. trying to boot test-sru-2 created with original packages resulted in booting to emergency mode.

tags: added: verification-done verification-done-impish
removed: verification-needed-impish
Revision history for this message
Nicholas Harvey (mobile-harvey) wrote (last edit ):

I can also confirm this proposed update fixes my scenario I reports in duplicate bug 1959522. Many thanks.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 2.0.6-1ubuntu2.1

---------------
zfs-linux (2.0.6-1ubuntu2.1) impish; urgency=medium

  * Fix uid function in zfs initramfs script. LP: #1946808

 -- Dimitri John Ledkov <email address hidden> Wed, 13 Oct 2021 11:12:37 +0100

Changed in zfs-linux (Ubuntu Impish):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for zfs-linux has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.