Update of libc6 packages breaks graphics driver intel-i915-dkms

[Mike Ferreira]

Installed and recent system updates of packages:
libc6-dbg/jammy-updates,now 2.35-0ubuntu3.3 amd64 [installed,automatic]
libc6-dev/jammy-updates,now 2.35-0ubuntu3.3 amd64 [installed,automatic]
libc6/jammy-updates,now 2.35-0ubuntu3.3 amd64 [installed,automatic]
libc6/jammy-updates,now 2.35-0ubuntu3.3 i386 [installed,automatic]

After updates, third party Iris XE Grahics driver package intel-i915-dkms breaks. Says kernel release parse failure.

Worked fine before this update.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: libc6 2.35-0ubuntu3.3
ProcVersionSignature: Ubuntu 6.2.0-32.32~22.04.1-generic 6.2.16
Uname: Linux 6.2.0-32-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair nvidia_modeset nvidia
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: pass
CloudArchitecture: x86_64
CloudID: none
CloudName: none
CloudPlatform: none
CloudSubPlatform: config
CurrentDesktop: GNOME
Date: Mon Sep 18 11:05:42 2023
InstallationDate: Installed on 2022-09-19 (364 days ago)
InstallationMedia: Ubuntu-Server 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220809)
ProcEnviron:
 LANGUAGE=en
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
RebootRequiredPkgs: Error: path contained symlinks.
SourcePackage: glibc
UpgradeStatus: No upgrade log present (probably fresh install)

[Mike Ferreira (mafoelffen)]

Ubuntu 22.04.3 LTS, Kernel 6.2.0-32.

Intel -9-13900K.

Attached crash Report.

[Mike Ferreira (mafoelffen)]

I know that package intel-i915-dkms is not an Ubuntu Package, but scheduled Ubuntu system updates of libc6 are what broke this driver.

Not actually sure what to do.

On the upstream Intel side, I have started a report to Intel. Will provide a link to that (here) when I finish it. In that, I will link back to this. (https://community.intel.com/t5/Graphics/Ubuntu-intel-i915-kdms-error-on-update/td-p/1524576)

It has to be somewhere in-between the two, so I thought I would cover both bases.

As I support Ubuntu users for the Linux graphics layer and Ubuntu hosted visualizations, this seems to be a good test case on the resolution of this.

[Mike Ferreira (mafoelffen)]

Attached UbuntuForums 'system-info' report to give and idea of the hardware and system at: https://paste.ubuntu.com/p/9FY5XgWX4d/

Code:
mafoelffen@msi-ubuntu:~$ apt list libc6* --installed
Listing... Done
libc6-dbg/jammy-updates,now 2.35-0ubuntu3.3 amd64 [installed,automatic]
libc6-dev/jammy-updates,now 2.35-0ubuntu3.3 amd64 [installed,automatic]
libc6/jammy-updates,now 2.35-0ubuntu3.3 amd64 [installed,automatic]
libc6/jammy-updates,now 2.35-0ubuntu3.3 i386 [installed,automatic]

[Simon Chopin (schopin)]

Hi, thanks for the report.

I was able to reproduce what seems to be the symptoms on a fresh VM with the previous version of glibc:

schopin@ubuntu2204:~$ cat /var/lib/dkms/intel-i915-dkms/1.23.5.19.230406.21.5.17.0.1034/build/make.log
DKMS make.log for intel-i915-dkms-1.23.5.19.230406.21.5.17.0.1034 for kernel 6.2.0-33-generic (x86_64)
mar. 19 sept. 2023 10:09:41 CEST
Generating local configuration database from kernel ...Kernel version parse failed!
make: *** [Makefile:47 : olddefconfig] Erreur 1
Generating local configuration database from kernel ...Kernel version parse failed!
make[1]: *** [Makefile:47 : modules] Erreur 1
make: *** [Makefile:34 : default] Erreur 2
schopin@ubuntu2204:~$ LANG=C dpkg -l libc6
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-===============-============-=================================
ii libc6:amd64 2.35-0ubuntu3.1 amd64 GNU C Library: Shared libraries

This should be investigated by the Intel team, as the failure really doesn't tell us much about what happened. Marking as Invalid as evidently the glibc SRU isn't at fault.

[Mike Ferreira (mafoelffen)]

If you look at the Link to Intel, their Internal Support Team is looking at it now.

I know this is an upstream issue with their code. But I figure the left hand needs to know what the right hand is doing, and I would document it so we "all" know what affects each other.

I will update this when we know is it resolved, as it affects Ubuntu Users using their i915 drivers.

[Mike Ferreira (mafoelffen)]

New Intel Case Number: 05993346

[Mike Ferreira (mafoelffen)]

Status Update:

The Intel Graphics Team's original suggested work-around suggested (https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-jammy-arc.html) was for Ubuntu 20.04 and downgrading to the 20.04 OEM Kernel series for that... 5.17.0-1020-oem

Their current updated installation guide (https://dgpu-docs.intel.com/driver/client/overview.html) is for 22.04 and states that it needs a minimum kernel 5.19.0 and better if on the 23.04 series of kernels.

With the upstream upgrade of GLIBC 2.37 (updated on 2023-09-12), intel-i915-dkms fails on any 6.2 and higher series of kernels.

I downgraded 22.04 to it's OEM series, which is at 6.1.0-1020-oem. That fixes the intel-i915-dkms crash, but it still fails on on current libva, so still is broken as any 'working' work-around.

I added a tickler to the Intel Graphics Team that 23.10 releases in 3 weeks with kernel 6.5 and GLIBC 2.38, and that 24.04 releases in less than 7 months.

What Intel seems to be over-looking, is that this all worked with the current kernel series before the recent updates on GLIBC 2.37.

I know this is all their upstream code, and I am actively working with them, trying to get this working for current Ubuntu and it's Users.

That is where this is at at the moment.

[Mike Ferreira (mafoelffen)]

Status Update: Intel Support Engineers have replicated the crash and have assured me (today) that they are working on it.

[Mike Ferreira (mafoelffen)]

Status Update: The new updated target date from the Intel Graphics Support Team is 2023.10.26. They are still working on it.

[Mike Ferreira (mafoelffen)]

IDK:
>>>
Try to remove the i915 dkms driver with cmd: “sudo apt remove -y intel-i915-dkms”.
Also you could try to install the dkms driver again. Latest release driver in repo already support 6.2.0 kernel.
If you still have issues with upper method, please provide '/var/log/syslog' file.
>>>
Even though they say it works with kernel 6.2.0, it does not (still).

Have gotten nowhere with this, and this is now almost 4 months.

I doubt any of this will work for Noble.

[Jianshui Yu (jianshui)]

Which platform are you using?
I installed the Ubuntu 22.04 and update to the latest packages via "sudo apt upgrade", the ARC GPU are running well already. and based on my understanding, Ubuntu22.04.6 already support ARC GPU and no need to install extra packages. if you need the latest drivers released from intel, only mesa/neo/media drivers are good enough, no need to install intel-i915-dkms.
It will only take issues if any new kernel API coming and intel-i915-dkms will support the new kernel several months later.

[Mike Ferreira (mafoelffen)]

Colin King asked me if he could look at it for me, but he's on vacation until January...

I was told this morning that the problem was when the LTS 22.04.3 point release was released. That's when Ubuntu changed it's LTS kernel series to 6.2.0-x with the HWE stack, that broke the dkms package they had in their Intel repo for jammy... (That is 'what' I reported back in September).

That (from @knayana-gse):
>>>
intel-i915-dkms 1.23.6.24.230425.29+i44-1, this package doesn't have 6.2 support.
Use latest dkms package: intel-i915-dkms/1.23.6.37.230425.49.
>>>

Unfortunately, that package version is not in their jammy repo:
>>>
mafoelffen@msi-ubuntu:~$ sudo apt list intel-i915-dkms -a
Listing... Done
intel-i915-dkms/unknown,unknown 1.23.8.20.230810.22+i33-1 all
intel-i915-dkms/unknown,unknown 1.23.7.17.230608.24+i37-1 all
intel-i915-dkms/unknown,unknown 1.23.6.24.230425.29+i44-1 all
intel-i915-dkms/unknown,unknown 1.23.5.19.230406.21.5.17.0.1034+i38-1 all
>>>
So that version of that package must be in another of their repo's(?)

If I can download it, if they don't do it themselves, I told then I would unpack it and try to patch it, and repackage it, to work with jammy's kernels.

There is now two Intel Support teams working on this. They told me to continue to try to get this working on their Support Forum (linked above), with the other team (also) following the progress there.

@jianshui
Platform is 22.04.3 on i9-13900K, info is noted above. Is iGPU, not external ARC GPU. Is not a GPU problem, is confirmed as kernel series version on 22.04.3 versus the intel-i915-dkms package version they have in their jammy repo.

This is not the vanilla in-kernel i915, but rather the datacenter/client modules that are meant to replace the abandoned Intel GVT-g for virtual shared passthroughs to VM's.

RE: Ubuntu 22.04 HWE kernel plus Intel OOT GPU driver --> https://dgpu-docs.intel.com/driver/client/overview.html#client-install-options

[Mike Ferreira (mafoelffen)]

I could just take a short-cut and save myself the work...

I'm one of the lead testers for Dev-Noble. I could just upgrade that machine as a test-case for that...

BUT:
That would close off what needs to be corrected for users to use that for Jammy, which still has official LTS support through April 2027. That is a long time to have other users hang on that.

As I also support Server Edition, and server customers usually do their production migrations after things are ironed out with our releases, I don't think that is the right thing to do. I'll stick with it until it is back-ported.

[Jianshui Yu (jianshui)]

Hi
you are right, 23.6.37.230425.49 isn't in the rolling release branch, I checked the latest rolling release note, the latest one is already 1.23.8.20.230810.22+i33-1 and support 6.2 kernel, but don't support 6.5 yet. for the 6.5 support it will take some time to develop and validate.

[Mike Ferreira (mafoelffen)]

Okay... This is resolved and working now.

RE: community.intel.com/t5/Graphics/Ubuntu-intel-i915-kdms-error-on-update/m-p/1556540#M126377

The solution was a brand new package that showed up in the Intel Repo this morning!!!

>>>
apt install -y intel-i915-dkms=1.23.9.11.231003.15+i19-1
>>>

I told them to "thank the elves" who worked on that package version last night in their workshop.

This now works for Intel OOT i915 for KVM shared i915 resources. I think that is a big win for Intel.

This will not work for Mantic and beyond yet (kernels 6.5.0-x). No word when and if that would be possible yet. Nor if they would even suggest that. I did ask though, so we might know what to expect.

[Jianshui Yu (jianshui)]

6.5 support is planned and now in the backlog, it could be ready after the new year in the fastest case.

[Mike Ferreira (mafoelffen)]

@jianshui -- Thank you. Can you please send me a message or email when it is ready for testing, so I can add them to my test-cases for DEV-Noble?

[Jianshui Yu (jianshui)]

np, will follow up with release manager.

[Mike Ferreira (mafoelffen)]

Unfortunately, that last update to intel-i916-dkms has a fatal problem that breaks zfs-linux: zed, with kernels 6.2.0-x.

All tests were successful... Or so I thought. Everything (Graphics related) worked. OOT modules built, VM's saw the iGPU of the host. Everything worked until the machine was powered down. Then would no boot.

This machine is ZFS-On-Root. The only pool that was still imported was rpool. All 4 other pools: bpool: /boot; hpool: /home; dpool: data; kpool: various datasets fro KVM... were not imported on the boot, zed would not come up. /dev/zd0 didn't exist.

Booted fine on 5.15.0-x kernels.

Removed intel-i915-dkms. Everything booted fine again.

Asked upstream intel if they want to reopen the support case, or if they want me to file another new case...

Why? This one builds the modules. That part is fixed. It just doesn't work with zfs-linux: zed on those same kernels. Fixed a problem... I was so excited that that it actually worked... But caused another (fatal) problem.

In holding pattern with that install until I get word from them. Installed Noble alongside it, and another instance of Jammy, so I can continue my other test cases for the release. I don't want it to be "totally out" committed & waiting another 4 months.

Dang. LOL.

[Jianshui Yu (jianshui)]

could you please share how to reproduce the issue? ZFS is new for me, it's better to share how to setup the zfs pool

[Mike Ferreira (mafoelffen)]

Copied from what I posted at: https://community.intel.com/t5/Graphics/Ubuntu-intel-i915-kdms-error-on-update/td-p/1524576#

To reproduce, Install Ubuntu Desktop 22.04.3 LTS with ZFS as the install method (In the install Method Panel under advanced options). I have 5 pools (rpool, bpool, hpool, dpool, & kpool), but it will fail with just the default 2 pools (rpool & bpool). Reboot. Log in. Install the Intel OOT drivers. All seems to work... Albeit, a problem just with some of the 'libva' libraries just when you try to run vainfo. --> xpu-smi & clinfo seem to show everything is working okay. Notes, the default install of desktop is which the HWE stack installed. ensure you have both linux-image-generic & linux-image-gerneric-hwe-22.04 installed
>>> apt list linux-image-generic-* --installed
For test purposes, so that you can get to the Grub2 menu, edit the /etc/default/grub file with these edits
>>> GRUB_TIMEOUT_STYLE=menu
>>> GRUB_TIMEOUT=5
>>> GRUB_RECORDFAIL_TIMEOUT=5
>>> GRUB_CMDLINE_LINUX_DEFAULT="--- --verbose debug drm.debug=0xe intel_iommu=on"
Save. Pick up the changes:
>>> sudo update-grub

Reboot...

With any kernel in the 6.2.0-X series: Will give an error saying something along the lines of "device /dev/zd0: file or device not found"... Then continue booting, taking a long time on udisk, the ending up at a root maintenance prompt. (Instead of the desktop). Hit Enter Key to go on.

At the root prompt, if you do
>>> zpool list
It will show that only rpool is imported. (bpool is not present) If you run
>>> zpool import bpool
>>> zpool list
All are imported fine. No matter what you do, if you boot from the ... Will result in the same.

Why? If you do
>>> systemctl status zfs-zed.service --no-pager
'zed' is inactive. It never starts. It will not start manually, and will not restart. <-- On a normal boot sequence with ZFS, the zed service should start, and you should see it import of each pool in the status... That is not happening at all.

If you boot any of the 5.15.0 series kernels, it boots normally...

I installed the OOT drivers on the second Ubuntu 22.04 LTS installation that is installed on LVM2, with no ZFS... same machine, just different location on it. It boots fine on 6.2.0 kernels.

[Mike Ferreira (mafoelffen)]

In the Ubuntu 22.04.3 LTS installer, in the order of the installer steps:

UEFI, 8GB RAM Minimum...

1. At Grub2 Menu > Try or Install
2. Panel 1 > Install Ubuntu, English
3. Panel 2 > Language English
4. Panel 3 > Keyboard English (US), English (US)
5. Panel 4 > Updates and Other Software > Full Install, Download Updates, Install 3rd party
6. Panel 5 > Installation Type: Erase and Install > Advanced Features > "Erase Disk and Use ZFS"
7. Confirm Partitioning
8. Choose Time Zone for Locale.
9. UserData > Keep going through installer for the rest of the UserData...

[Mike Ferreira (mafoelffen)]

@jianshui ---

Just so you know 22.04 LTS just went to Linux Kernel 6.5.0-14 in updates... Once you apply updates, you are going to see that. So LTS with the HWE stack is now in the 6.5 series.

What does that mean to Intel and us with this?

I no longer thing you need to worry about Linux Kernel 6.2.0 series. I no longer think it's going to matter for that 6.2.0 series now(?) for 22.04.3 and newer...

[Elena Zhou (jinghuaz)]

@Mike

Hi we have just validated on Ubuntu 22.04 with kernel 6.2.0-26 but the issue cannot be reproduced.
Are there any steps missing?
I have created 5 pools in 2 hard disk (including rpool and bpool); Installed dkms driver; Reboot;
And I can't see any wrong msgs in kern.log about /dev/zd0 and zd*
By using "zpool list", the 5 pools shows.
Are there any specific variables in your steps?
How do you create 3 pools?

[Mike Ferreira (mafoelffen)]

Not really important to reproduces more than just the basic rpool, and bpool, as it should get the same error when it does find rpool, then bpool, but...

On that workstation, I have ZFS pools rpool, bpool, dpool, kpool, & hpool:
>>>
mafoelffen@msi-ubuntu:~$ sudo zfs list
[sudo] password for mafoelffen:
NAME USED AVAIL REFER MOUNTPOINT
bpool 707M 1.93G 192K /boot
bpool/BOOT 705M 1.93G 192K none
bpool/BOOT/ubuntu_2204 705M 1.93G 705M /boot
dpool 2.10M 450G 96K /dpool
dpool/DATA 192K 450G 96K none
dpool/DATA/ubuntu_2204 96K 450G 96K /dpool
hpool 2.75G 344G 192K /home
hpool/USERDATA 2.73G 344G 192K none
hpool/USERDATA/ubuntu_2204 2.73G 344G 2.73G /home
kpool 429G 470G 96K /kpool
kpool/QEMU-KVM 429G 470G 96K none
kpool/QEMU-KVM/ubuntu_2204 429G 470G 104K /kpool
kpool/QEMU-KVM/ubuntu_2204/ISO 411G 470G 411G /kpool/ISO
kpool/QEMU-KVM/ubuntu_2204/KVM-VM 96K 470G 96K /kpool/KVM-VM
kpool/QEMU-KVM/ubuntu_2204/images 18.3G 470G 18.3G /var/lib/libvirt/images
kpool/QEMU-KVM/ubuntu_2204/libvirt 648K 470G 648K /etc/libvirt/
rpool 22.3G 702G 192K /
rpool/ROOT 22.2G 702G 192K none
rpool/ROOT/ubuntu_2204 22.2G 702G 11.8G /
rpool/ROOT/ubuntu_2204/srv 192K 702G 192K /srv
rpool/ROOT/ubuntu_2204/usr 496K 702G 192K /usr
rpool/ROOT/ubuntu_2204/usr/local 304K 702G 304K /usr/local
rpool/ROOT/ubuntu_2204/var 10.4G 702G 192K /var
rpool/ROOT/ubuntu_2204/var/cache 467M 702G 467M /var/cache
rpool/ROOT/ubuntu_2204/var/games 192K 702G 192K /var/games
rpool/ROOT/ubuntu_2204/var/lib 7.89G 702G 7.69G /var/lib
rpool/ROOT/ubuntu_2204/var/lib/AccountsService 240K 702G 240K /var/lib/AccountsService
rpool/ROOT/ubuntu_2204/var/lib/NetworkManager 296K 702G 296K /var/lib/NetworkManager
rpool/ROOT/ubuntu_2204/var/lib/apt 111M 702G 111M /var/lib/apt
rpool/ROOT/ubuntu_2204/var/lib/docker 192K 702G 192K /var/lib/docker
rpool/ROOT/ubuntu_2204/var/lib/dpkg 93.3M 702G 93.3M /var/lib/dpkg
rpool/ROOT/ubuntu_2204/var/lib/nfs 192K 702G 192K /var/lib/nfs
rpool/ROOT/ubuntu_2204/var/log 2.07G 702G 2.07G /var/log
rpool/ROOT/ubuntu_2204/var/mail 192K 702G 192K /var/mail
rpool/ROOT/ubuntu_2204/var/snap ...

[Mike Ferreira (mafoelffen)]

Here is the error from the boot logs:
>>>
mafoelffen@msi-ubuntu:~$ sudo grep -m2 -A30 -E '/dev/zvol' /var/log/boot.log.7
find: /dev/zvol/: No such file or directory
find: /dev/zvol/: No such file or directory
[FAILED] Failed to start Wait for udev To Complete Device Initialization.
[DEPEND] Dependency failed for Import ZFS pools by cache file.
[DEPEND] Dependency failed for Install ZFS kernel module.
[FAILED] Failed to mount /boot.
[DEPEND] Dependency failed for /boot/grub.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Unattended Upgrades Shutdown.
[DEPEND] Dependency failed for /boot/efi.
[FAILED] Failed to mount /dpool.
[FAILED] Failed to mount /etc/libvirt.
[FAILED] Failed to mount /home.
[DEPEND] Dependency failed for /home/mafoelffen/WIN_G.
[FAILED] Failed to mount /kpool.
[DEPEND] Dependency failed for /kpool/ISO.
[DEPEND] Dependency failed for /kpool/KVM-VM.
[FAILED] Failed to mount /var/lib/libvirt/images.
[FAILED] Failed to start Set Up Additional Binary Formats.
[ OK ] Finished Mount ZFS filesystems.
[ OK ] Reached target Mounting snaps.
         Mounting Mount unit for bare, revision 5...
         Mounting Mount unit for canonical-livepatch, revision 246...
         Mounting Mount unit for canonical-livepatch, revision 248...
         Mounting Mount unit for core, revision 16091...
         Mounting Mount unit for core, revision 16202...
         Mounting Mount unit for core20, revision 2015...
         Mounting Mount unit for core20, revision 2105...
         Mounting Mount unit for core22, revision 1033...
         Mounting Mount unit for core22, revision 864...
         Mounting Mount unit for firefox, revision 3358...
         Mounting Mount unit for firefox, revision 3416...
>>>
But remember at the end of that, it dumps down to a root maintenance prompt, where I can manually import the pools and everything is there.

[Mike Ferreira (mafoelffen)]

From dmesg:
>>>
mafoelffen@msi-ubuntu:~$ zgrep -a 'zvol' /var/log/syslog.{1,2,3,4}.gz | grep -v 'os-probes'
gzip: /var/log/syslog.1.gz: No such file or directory
/var/log/syslog.2.gz:Jan 18 08:09:45 msi-ubuntu systemd[1]: Starting Wait for ZFS Volume (zvol) links in /dev...
/var/log/syslog.2.gz:Jan 18 08:09:45 msi-ubuntu zvol_wait[3560]: No zvols found, nothing to do.
/var/log/syslog.2.gz:Jan 18 08:09:45 msi-ubuntu systemd[1]: Finished Wait for ZFS Volume (zvol) links in /dev.
/var/log/syslog.3.gz:Jan 10 22:34:17 msi-ubuntu systemd[1]: Starting Wait for ZFS Volume (zvol) links in /dev...
/var/log/syslog.3.gz:Jan 10 22:34:17 msi-ubuntu zvol_wait[3561]: No zvols found, nothing to do.
/var/log/syslog.3.gz:Jan 10 22:34:17 msi-ubuntu systemd[1]: Finished Wait for ZFS Volume (zvol) links in /dev.
/var/log/syslog.4.gz:Dec 31 07:23:15 msi-ubuntu systemd[1]: Starting Wait for ZFS Volume (zvol) links in /dev...
/var/log/syslog.4.gz:Dec 31 07:23:15 msi-ubuntu zvol_wait[3909]: No zvols found, nothing to do.
/var/log/syslog.4.gz:Dec 31 07:23:15 msi-ubuntu systemd[1]: Finished Wait for ZFS Volume (zvol) links in /dev.
/var/log/syslog.4.gz:Jan 1 23:52:31 msi-ubuntu systemd[1]: Starting Wait for ZFS Volume (zvol) links in /dev...
/var/log/syslog.4.gz:Jan 1 23:52:31 msi-ubuntu zvol_wait[4068]: No zvols found, nothing to do.
/var/log/syslog.4.gz:Jan 1 23:52:31 msi-ubuntu systemd[1]: Finished Wait for ZFS Volume (zvol) links in /dev.
/var/log/syslog.4.gz:Jan 2 20:03:32 msi-ubuntu systemd[1]: Starting Wait for ZFS Volume (zvol) links in /dev...
/var/log/syslog.4.gz:Jan 2 20:03:32 msi-ubuntu zvol_wait[4148]: No zvols found, nothing to do.
/var/log/syslog.4.gz:Jan 2 20:03:32 msi-ubuntu systemd[1]: Finished Wait for ZFS Volume (zvol) links in /dev.

>>>

[Mike Ferreira (mafoelffen)]

In the syslog snippet attached above in the last comment, look at Dec 29 10:32:51 on...

[Elena Zhou (jinghuaz)]

All the logs in my machine look fine (below):
Maybe there are some important steps we missed?

1. zvol related like this:

/var/log/syslog.2.gz:Jan 25 02:22:29 adl-02 systemd[3952]: Got message type=method_call sender=:1.39 destination=:1.0 path=/org/freedesktop/systemd1/unit/dev_2dzvol_2dpool_5f6_2ddisk7_5cx2dpart1_2edevice interface=org.freedesktop.DBus.Introspectable member=Introspect cookie=903 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
/var/log/syslog.2.gz:Jan 25 02:22:29 adl-02 systemd[3952]: Got message type=method_call sender=:1.39 destination=:1.0 path=/org/freedesktop/systemd1/unit/dev_2dzvol_2dpool_5f6_2ddisk7_2edevice interface=org.freedesktop.DBus.Introspectable member=Introspect cookie=1037 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
/var/log/syslog.2.gz:Jan 25 02:22:29 adl-02 systemd[3952]: Got message type=method_call sender=:1.39 destination=:1.0 path=/org/freedesktop/systemd1/unit/dev_2dzvol_2drpool_2ddisk1_2edevice interface=org.freedesktop.DBus.Introspectable member=Introspect cookie=1061 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
gzip: /var/log/syslog.3.gz: No such file or directory

2. zvol related:

gta@adl-02:~$ sudo grep -m2 -A30 -E '/dev/zvol' /var/log/boot.log.5
find: /dev/zvol/: No such file or directory
find: /dev/zvol/: No such file or directory
[ OK ] Finished Tell Plymouth To Write Out Runtime Data.
[ OK ] Finished Record System Boot/Shutdown in UTMP.
[ OK ] Started Network Time Synchronization.
[ OK ] Reached target System Time Set.
[ OK ] Started Userspace Out-Of-Memory (OOM) Killer.
[ OK ] Started Network Name Resolution.
[ OK ] Reached target Host and Network Name Lookups.
[ OK ] Finished Load AppArmor profiles.
         Starting Load AppArmor profiles managed internally by snapd...
[ OK ] Finished Load AppArmor profiles managed internally by snapd.
[ OK ] Reached target System Initialization.
[ OK ] Started ACPI Events Check.
[ OK ] Started CUPS Scheduler.
[ OK ] Started Start whoopsie on modification of the /var/crash directory.
[ OK ] Started Trigger anacron every hour.
[ OK ] Started Daily apt download activities.
[ OK ] Started Daily apt upgrade and clean activities.
[ OK ] Started Daily dpkg database backup timer.
[ OK ] Started Periodic ext4 Online Metadata Check for All Filesystems.
[ OK ] Started Discard unused blocks once a week.
[ OK ] Started Refresh fwupd metadata regularly.
[ OK ] Started Daily rotation of log files.
[ OK ] Started Daily man-db regeneration.
[ OK ] Started Message of the Day.
[ OK ] Started Daily Cleanup of Temporary Directories.
[ OK ] Reached target Path Units.
[ OK ] Listening on ACPID Listen Socket.
[ OK ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
[ OK ] Listening on CUPS Scheduler.
[ OK ] Listening on D-Bus System Message Bus Socket.

3. zfs list related:

gta@adl-02:~$ sudo zfs list
NAME USED AVAIL REFER MOUNTPOINT
bpool 223M 1.53G 96K /boot
bpool/BOOT 222M 1.53G 96K none
bpool/BOOT/ubuntu_...