BUG: kernel NULL pointer dereference, address: 00000000000006c8

[Lars]

I am trying to get rid of the bug#1906476, so I installed a new Ubuntu/Impish on another Disk with an unencrypted rpool followed the tutorial at
https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2020.04%20Root%20on%20ZFS.html

After installation I copied the most from the encrypted disk to the unencrypted one via booting from stick, importing encrypted rpool and unencrypted rpool and using rsync.
The first reboot into the new disk hits another Bug:

[ 0.126013] Kernel command line: BOOT_IMAGE=/BOOT/ubuntu_1m79i1@/vmlinuz-5.13.0-23-generic root=ZFS=rpool_unencrypted/ROOT/ubuntu_1m79i1 ro text console=tty0 console=ttyS0,115200n8 nosplash init_on_alloc=0
...
Begin: Running /scripts/local-premount ... done.
[ 66.651432] spl: loading out-of-tree module taints kernel.
[ 66.666011] znvpair: module license 'CDDL' taints kernel.
[ 66.673672] Disabling lock debugging due to kernel taint
[ 66.849345] ZFS: Loaded module v2.0.6-1ubuntu2, ZFS pool version 5000, ZFS filesystem version 5
Begin: Importing ZFS root pool 'rpool_unencrypted' ... [ 67.920948] BUG: kernel NULL pointer dereference, address: 00000000000006c8
[ 67.929514] #PF: supervisor write access in kernel mode
[ 67.935991] #PF: error_code(0x0002) - not-present page
[ 67.942355] PGD 0 P4D 0
[ 67.946127] Oops: 0002 [#1] SMP PTI
[ 67.950831] CPU: 5 PID: 341 Comm: zpool Tainted: P O 5.13.0-23-generic #23-Ubuntu
[ 67.960839] Hardware name: Sun Microsystems SUN FIRE X4150/SUN FIRE X4150, BIOS 1ADQW068 11/16/2010
[ 67.971112] RIP: 0010:mutex_lock+0x1e/0x40
[ 67.976427] Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc e8 cd ec ff ff 31 c0 65 48 8b 14 25 c0 7b 01 00 <f0> 49 0f b1 14 24 75 06 4c 8b 65 f8 c9 c3 4c 89 e7 e8 ac ff ff ff
[ 67.997696] RSP: 0018:ffffb7a0c0897b08 EFLAGS: 00010246
[ 68.004175] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 68.012565] RDX: ffff8d9654f49700 RSI: 0000000000000000 RDI: 00000000000006c8
[ 68.020937] RBP: ffffb7a0c0897b10 R08: ffff8d9642091d20 R09: ffff8d9642091d20
[ 68.029298] R10: ffff8d9656462400 R11: ffff8d9656462400 R12: 00000000000006c8
[ 68.037658] R13: ffffffffc0841458 R14: 00000000000006e8 R15: 0000000000000000
[ 68.046023] FS: 00007f9315e5f7c0(0000) GS:ffff8d9853d40000(0000) knlGS:0000000000000000
[ 68.055341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 68.062313] CR2: 00000000000006c8 CR3: 0000000115f48000 CR4: 00000000000006e0
[ 68.070671] Call Trace:
[ 68.074330] rrw_enter_read_impl+0x22/0x100 [zfs]
[ 68.080520] rrw_enter_read+0x13/0x20 [zfs]
[ 68.086096] rrw_enter+0x1d/0x20 [zfs]
[ 68.091202] dsl_pool_config_enter+0x1d/0x20 [zfs]
[ 68.097328] spa_prop_get+0x98/0x3c0 [zfs]
[ 68.102750] ? spl_kmem_free_impl+0x25/0x30 [spl]
[ 68.108601] ? __raw_callee_save___native_queued_spin_unlock+0x15/0x23
[ 68.116274] ? __raw_callee_save___native_queued_spin_unlock+0x15/0x23
[ 68.123923] ? __raw_callee_save___native_queued_spin_unlock+0x15/0x23
[ 68.131555] ? queued_spin_unlock+0x9/0x10 [zfs]
[ 68.137462] ? do_raw_spin_unlock+0x9/0x10 [zfs]
[ 68.143346] ? __raw_spin_unlock+0x9/0x10 [zfs]
[ 68.149140] ? spa_open_common+0x4fd/0x510 [zfs]
[ 68.155013] ? spa_name_compare+0xe/0x30 [zfs]
[ 68.160704] ? avl_find+0x5f/0x90 [zavl]
[ 68.165666] zfs_ioc_pool_get_props+0x79/0x140 [zfs]
[ 68.171864] zfsdev_ioctl_common+0x645/0x6f0 [zfs]
[ 68.177890] ? __check_object_size.part.0+0x4a/0x150
[ 68.183897] ? _copy_from_user+0x2e/0x60
[ 68.188852] zfsdev_ioctl+0x57/0xe0 [zfs]
[ 68.194081] __x64_sys_ioctl+0x91/0xc0
[ 68.198864] do_syscall_64+0x61/0xb0
[ 68.203470] ? handle_mm_fault+0xda/0x2c0
[ 68.208514] ? do_user_addr_fault+0x1d0/0x660
[ 68.213899] ? exit_to_user_mode_prepare+0x37/0xb0
[ 68.219719] ? irqentry_exit_to_user_mode+0x9/0x20
[ 68.225536] ? irqentry_exit+0x19/0x30
[ 68.230279] ? exc_page_fault+0x8f/0x170
[ 68.235169] ? asm_exc_page_fault+0x8/0x30
[ 68.240225] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 68.246237] RIP: 0033:0x7f931644f9cb
[ 68.250769] Code: ff ff ff 85 c0 79 8b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 35 a4 0f 00 f7 d8 64 89 01 48
[ 68.271548] RSP: 002b:00007fff7de60648 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 68.280137] RAX: ffffffffffffffda RBX: 0000555b30942ca0 RCX: 00007f931644f9cb
[ 68.288279] RDX: 00007fff7de60650 RSI: 0000000000005a27 RDI: 0000000000000003
[ 68.296398] RBP: 00007fff7de63c30 R08: 0000555b30955f80 R09: 00007f931654b3d0
[ 68.304489] R10: fffffffffffff000 R11: 0000000000000246 R12: 00007fff7de60650
[ 68.312560] R13: 0000555b3093b320 R14: 0000000000000000 R15: 0000555b30942ca0
[ 68.320627] Modules linked in: zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ses enclosure scsi_transport_sas uas usb_storage hid_generic ast usbhid drm_vram_helper i2c_algo_bit hid drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec rc_core gpio_ich i2c_i801 ahci drm psmouse libahci pata_acpi i2c_smbus lpc_ich aacraid e1000e
[ 68.359784] CR2: 00000000000006c8
[ 68.364158] ---[ end trace 8157c208a863deb0 ]---
[ 68.369571] RIP: 0010:mutex_lock+0x1e/0x40
[ 68.374431] Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc e8 cd ec ff ff 31 c0 65 48 8b 14 25 c0 7b 01 00 <f0> 49 0f b1 14 24 75 06 4c 8b 65 f8 c9 c3 4c 89 e7 e8 ac ff ff ff
[ 68.394593] RSP: 0018:ffffb7a0c0897b08 EFLAGS: 00010246
[ 68.400538] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 68.408402] RDX: ffff8d9654f49700 RSI: 0000000000000000 RDI: 00000000000006c8
[ 68.416265] RBP: ffffb7a0c0897b10 R08: ffff8d9642091d20 R09: ffff8d9642091d20
[ 68.424130] R10: ffff8d9656462400 R11: ffff8d9656462400 R12: 00000000000006c8
[ 68.431999] R13: ffffffffc0841458 R14: 00000000000006e8 R15: 0000000000000000
[ 68.439862] FS: 00007f9315e5f7c0(0000) GS:ffff8d9853d40000(0000) knlGS:0000000000000000
[ 68.448702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 68.455211] CR2: 00000000000006c8 CR3: 0000000115f48000 CR4: 00000000000006e0
Killed

[Lars (lollypop)]

Updating to jammy fixed this problem...

-> Close

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Lars (lollypop)]

Oops I did it again...

Hi folks,
after moving my zpool to another hardware because of a broken system I got almost the same error with jammy. So it might not be fixed and I still have a problem.

[ 48.494682] ZFS: Loaded module v2.1.2-1ubuntu3, ZFS pool version 5000, ZFS filesystem version 5
Begin: Importing ZFS root pool 'rpool_unencrypted' ... [ 51.538634] BUG: kernel NULL pointer dereference, address: 00000000000006c8
[ 51.545731] #PF: supervisor write access in kernel mode
[ 51.551043] #PF: error_code(0x0002) - not-present page
[ 51.556313] PGD 0 P4D 0
[ 51.558952] Oops: 0002 [#1] SMP PTI
[ 51.562543] CPU: 4 PID: 665 Comm: zpool Tainted: P O 5.15.0-39-generic #42-Ubuntu
[ 51.571448] Hardware name: Oracle Corporation SUN FIRE X4170 M3 /MOTHER BOARD ASSEMBL , BIOS 17160400 05/06/2020
[ 51.582183] RIP: 0010:mutex_lock+0x1e/0x40
[ 51.586384] Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc e8 2d ed ff ff 31 c0 65 48 8b 14 25 c0 fb 01 00 <f0> 49 0f b1 14 24 75 06 4c 8b 65 f8 c9 c3 4c 89 e7 e8 ac ff ff ff
[ 51.605298] RSP: 0018:ffffb0128eb03b40 EFLAGS: 00010246
[ 51.610628] RAX: 0000000000000000 RBX: ffff9c23604ac000 RCX: 0000000000000000
[ 51.617868] RDX: ffff9c2361613180 RSI: 0000000000000000 RDI: 00000000000006c8
[ 51.625106] RBP: ffffb0128eb03b48 R08: ffff9c2347e5cda0 R09: ffff9c2347e5cda0
[ 51.632358] R10: 0000000000000001 R11: 0000000000000000 R12: 00000000000006c8
[ 51.639594] R13: 0000000000000000 R14: 00000000000006c8 R15: 00000000000006e8
[ 51.646831] FS: 00007ff929d1e7c0(0000) GS:ffff9c3a5fb00000(0000) knlGS:0000000000000000
[ 51.655035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 51.660878] CR2: 00000000000006c8 CR3: 0000000123c1e006 CR4: 00000000000606e0
[ 51.669603] Call Trace:
[ 51.672157] <TASK>
[ 51.674377] rrw_enter_read_impl+0x29/0x110 [zfs]
[ 51.679459] rrw_enter_read+0x13/0x20 [zfs]
[ 51.683981] rrw_enter+0x1d/0x20 [zfs]
[ 51.688070] dsl_pool_config_enter+0x1d/0x20 [zfs]
[ 51.693186] spa_prop_get+0x98/0x3c0 [zfs]
[ 51.697624] ? spl_kmem_free_impl+0x25/0x30 [spl]
[ 51.702441] ? kfree+0x216/0x250
[ 51.705782] ? __cond_resched+0x1a/0x50
[ 51.709727] ? avl_destroy_nodes+0x9e/0xf0 [zavl]
[ 51.714522] ? __cond_resched+0x1a/0x50
[ 51.718445] ? do_raw_spin_unlock+0x9/0x10 [zfs]
[ 51.723407] ? __raw_spin_unlock+0x9/0x10 [zfs]
[ 51.728307] ? spa_deactivate+0x22b/0x320 [zfs]
[ 51.733181] ? do_raw_spin_unlock+0x9/0x10 [zfs]
[ 51.738138] ? __raw_spin_unlock+0x9/0x10 [zfs]
[ 51.743009] ? spa_name_compare+0xe/0x30 [zfs]
[ 51.747801] ? avl_find+0x6b/0xd0 [zavl]
[ 51.751814] zfs_ioc_pool_get_props+0x79/0x140 [zfs]
[ 51.757145] zfsdev_ioctl_common+0x682/0x740 [zfs]
[ 51.762312] ? __check_object_size.part.0+0x4a/0x150
[ 51.767389] ? _copy_from_user+0x2e/0x60
[ 51.771427] zfsdev_ioctl+0x57/0xe0 [zfs]
[ 51.775802] __x64_sys_ioctl+0x91/0xc0
[ 51.779668] do_syscall_64+0x5c/0xc0
[ 51.783358] ? do_user_addr_fault+0x1e3/0x670
[ 51.787831] ? syscall_exit_to_user_mode+0x27/0x50
[ 51.792729] ? exit_to_user_mode_prepare+0x37/0xb0
[ 51.797635] ? irqentry_exit_to_us...

[Lars (lollypop)]

Bootin with boot option break=premount:
(initramfs) modprobe zfs
[ 45.484502] spl: loading out-of-tree module taints kernel.
[ 45.503302] icp: module license 'CDDL' taints kernel.
[ 45.509218] Disabling lock debugging due to kernel taint
[ 45.709339] ZFS: Loaded module v2.1.2-1ubuntu3, ZFS pool version 5000, ZFS filesystem version 5
(initramfs) zpool import
[ 48.811749] random: crng init done
   pool: rpool
     id: 396532089768583635
  state: ONLINE
status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
 the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
 config:

 rpool ONLINE
  sda4 ONLINE

   pool: bpool
     id: 3142261943073140017
  state: ONLINE
status: Some supported features are not enabled on the pool.
 (Note that they may be intentionally disabled if the
 'compatibility' property is set.)
 action: The pool can be imported using its name or numeric identifier, though
 some features will not be available without an explicit 'zpool upgrade'.
 config:

 bpool ONLINE
  sda3 ONLINE
(initramfs) zpool import -fN rpool
(initramfs) zpool import -fN bpool
(initramfs) zpool status
  pool: bpool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:55 with 0 errors on Mon Sep 12 19:50:53 2022
config:

 NAME STATE READ WRITE CKSUM
 bpool ONLINE 0 0 0
  sda3 ONLINE 0 0 0

errors: No known data errors
[ 84.038049] BUG: kernel NULL pointer dereference, address: 00000000000006c8
[ 84.046949] #PF: supervisor write access in kernel mode
[ 84.053988] #PF: error_code(0x0002) - not-present page
[ 84.060851] PGD 0 P4D 0
[ 84.065092] Oops: 0002 [#1] SMP PTI
[ 84.070315] CPU: 19 PID: 1636 Comm: zpool Tainted: P O 5.15.0-40-generic #43-Ubuntu
[ 84.081023] Hardware name: Oracle Corporation SUN FIRE X4170 M3 /MOTHER BOARD ASSEMBL , BIOS 17160400 05/06/2020
[ 84.093456] RIP: 0010:mutex_lock+0x1e/0x40
[ 84.099364] Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc e8 2d ed ff ff 31 c0 65 48 8b 14 25 c0 fb 01 00 <f0> 49 0f b1 14 24 75 06 4c 8b 65 f8 c9 c3 4c 89 e7 e8 ac ff ff ff
[ 84.121768] RSP: 0018:ffff9fc8b7747b08 EFLAGS: 00010246
[ 84.128829] RAX: 0000000000000000 RBX: ffff8eeaa36bc000 RCX: 0000000000000000
[ 84.137866] RDX: ffff8eeab81298c0 RSI: 0000000000000000 RDI: 00000000000006c8
[ 84.146854] RBP: ffff9fc8b7747b10 R08: ffff8eea87ce49e0 R09: ffff8eea87ce49e0
[ 84.155794] R10: 0000000000000001 R11: 0000000000000000 R12: 00000000000006c8
[ 84.164788] R13: 0000000000000000 R14: 00000000000006c8 R15: 00000000000006e8
[ 84.173762] FS: 00007f5cdf0407c0(0000) GS:ffff8f019fcc0000(0000) knlGS:0000000000000000
[ 84.183673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 84.191251] CR2: 00000000000006c8 CR3: 0000000120e28005 CR4: 00000000000606e0
[ 84.200216] Call Trace:
[ 84.204480] <TASK>
[ 84.208371] rrw_enter_read_impl+0x29/0x110 [zfs]
[ 84.215182] rrw_enter_read+0x13/0x20 [zfs]
[ 84.221387] rrw_enter+0x1d/0x20 [zfs]
[ 84.227149] dsl_pool_config_enter+0x1d/0x20 [zfs]
[ 84.2...

[Lars (lollypop)]

P.S.: I renamed the zpools from rpool_unencrypted to rpool and from bpool_unencrypted to bpool. Just to find out if this is the problem.

[Lars (lollypop)]

Next test:
Same with boot options break=premount, but then only bpool imported:
(initramfs) modprobe zfs
(initramfs) zpool import -N bpool
(initramfs) zpool status
  pool: bpool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:55 with 0 errors on Mon Sep 12 19:50:53 2022
config:

 NAME STATE READ WRITE CKSUM
 bpool ONLINE 0 0 0
  sda3 ONLINE 0 0 0

errors: No known data errors
[ 74.151238] BUG: kernel NULL pointer dereference, address: 00000000000006c8
[ 74.160142] #PF: supervisor write access in kernel mode
[ 74.167194] #PF: error_code(0x0002) - not-present page
[ 74.174128] PGD 0 P4D 0
[ 74.178425] Oops: 0002 [#1] SMP PTI
[ 74.183657] CPU: 21 PID: 1006 Comm: zpool Tainted: P O 5.15.0-40-generic #43-Ubuntu
[ 74.194395] Hardware name: Oracle Corporation SUN FIRE X4170 M3 /MOTHER BOARD ASSEMBL , BIOS 17160400 05/06/2020
[ 74.206842] RIP: 0010:mutex_lock+0x1e/0x40
[ 74.212748] Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc e8 2d ed ff ff 31 c0 65 48 8b 14 25 c0 fb 01 00 <f0> 49 0f b1 14 24 75 06 4c 8b 65 f8 c9 c3 4c 89 e7 e8 ac ff ff ff
[ 74.235223] RSP: 0018:ffffaa064ece3b58 EFLAGS: 00010246
[ 74.242334] RAX: 0000000000000000 RBX: ffff9d3aa27ec000 RCX: 0000000000000000
[ 74.251373] RDX: ffff9d3a9e8e3180 RSI: 0000000000000000 RDI: 00000000000006c8
[ 74.260384] RBP: ffffaa064ece3b60 R08: ffff9d3a87d626c0 R09: ffff9d3a87d626c0
[ 74.269383] R10: ffff9d3ab9a5c800 R11: 0000000000000000 R12: 00000000000006c8
[ 74.278371] R13: 0000000000000000 R14: 00000000000006c8 R15: 00000000000006e8
[ 74.287355] FS: 00007f3c219a67c0(0000) GS:ffff9d519fd40000(0000) knlGS:0000000000000000
[ 74.297326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 74.304922] CR2: 00000000000006c8 CR3: 00000001231e0003 CR4: 00000000000606e0
[ 74.313923] Call Trace:
[ 74.318223] <TASK>
[ 74.322154] rrw_enter_read_impl+0x29/0x110 [zfs]
[ 74.328957] rrw_enter_read+0x13/0x20 [zfs]
[ 74.335188] rrw_enter+0x1d/0x20 [zfs]
[ 74.340961] dsl_pool_config_enter+0x1d/0x20 [zfs]
[ 74.347762] spa_prop_get+0x98/0x3c0 [zfs]
[ 74.353891] ? spl_kmem_free_impl+0x25/0x30 [spl]
[ 74.360376] ? spl_kmem_free_impl+0x25/0x30 [spl]
[ 74.366838] ? kfree+0x1f3/0x250
[ 74.371805] ? __cond_resched+0x1a/0x50
[ 74.377357] ? avl_destroy_nodes+0x9e/0xf0 [zavl]
[ 74.383771] ? __cond_resched+0x1a/0x50
[ 74.389275] ? do_raw_spin_unlock+0x9/0x10 [zfs]
[ 74.395761] ? __raw_spin_unlock+0x9/0x10 [zfs]
[ 74.402143] ? spa_deactivate+0x22b/0x320 [zfs]
[ 74.408521] ? do_raw_spin_unlock+0x9/0x10 [zfs]
[ 74.414974] ? __raw_spin_unlock+0x9/0x10 [zfs]
[ 74.421340] ? spa_name_compare+0xe/0x30 [zfs]
[ 74.427646] ? avl_find+0x6b/0xd0 [zavl]
[ 74.433159] zfs_ioc_pool_get_props+0x79/0x140 [zfs]
[ 74.439986] zfsdev_ioctl_common+0x682/0x740 [zfs]
[ 74.446633] ? __check_object_size.part.0+0x4a/0x150
[ 74.453189] ? _copy_from_user+0x2e/0x60
[ 74.458693] zfsdev_ioctl+0x57/0xe0 [zfs]
[ 74.464552] __x64_sys_ioctl+0x91/0xc0
[ 74.469863] do_syscall_64+0x5c/0xc0
[ 74.474954] ? do_user_addr_fault+0x1e3/0x670
[ 74.480...