NULL pointer dereference in vdev_mirror_io_done

[Sevan Janiyan]

Ubuntu 22.10 riscv64 image on lichee rv board with 3 HDDs attached, 2 in a mirrored zpool, 1 standalone zpool, rsync copying data from single pool to mirrored pool.

[Sun Nov 13 06:41:10 2022] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000060
[Sun Nov 13 06:41:10 2022] Oops [#1]
[Sun Nov 13 06:41:10 2022] Modules linked in: binfmt_misc 8723ds(OE) cfg80211 zfs(POE) pwrseq_simple zunicode(POE) sunxi_cir rc_core snd_s
oc_hdmi_codec zzstd(OE) ses enclosure scsi_transport_sas sunxi phy_generic zlua(OE) sunxi_cedrus(C) zcommon(POE) v4l2_mem2mem videobuf2_dm
a_contig znvpair(POE) videobuf2_memops zavl(POE) videobuf2_v4l2 sun20i_codec snd_soc_simple_card snd_soc_simple_card_utils videobuf2_commo
n snd_soc_dmic icp(POE) leds_sun50i_r329 led_class_multicolor spl(OE) snd_soc_core videodev snd_compress ac97_bus mc snd_pcm_dmaengine snd
_pcm snd_timer snd leds_gpio soundcore nls_iso8859_1 uio_pdrv_genirq uio ramoops pstore_blk reed_solomon pstore_zone efi_pstore dm_multipa
th scsi_dh_rdac scsi_dh_emc scsi_dh_alua ip_tables x_tables autofs4 efivarfs raid10 raid456 libcrc32c async_raid6_recov async_memcpy async
_pq async_xor xor async_tx raid6_pq raid1 raid0 multipath linear dw_hdmi_i2s_audio dw_hdmi_cec sun4i_drm sun8i_drm_hdmi sun4i_frontend sun
8i_mixer dw_hdmi display_connector sun4i_tcon
[Sun Nov 13 06:41:10 2022] drm_cma_helper drm_kms_helper sun8i_tcon_top backlight syscopyarea sysfillrect sysimgblt fb_sys_fops
[Sun Nov 13 06:41:10 2022] CPU: 0 PID: 784 Comm: z_rd_int Tainted: P C OE 5.17.0-1003-allwinner #3-Ubuntu
[Sun Nov 13 06:41:10 2022] Hardware name: Sipeed Lichee RV Dock (DT)
[Sun Nov 13 06:41:10 2022] epc : zio_vdev_child_io+0x34/0x104 [zfs]
[Sun Nov 13 06:41:10 2022] ra : vdev_mirror_io_done+0x276/0x3a0 [zfs]
[Sun Nov 13 06:41:10 2022] epc : ffffffff03c5ad7a ra : ffffffff03c15c3e sp : ffffffd829c4bb90
[Sun Nov 13 06:41:10 2022] gp : ffffffff81e93b80 tp : ffffffd807918000 t0 : ffffffd7fffcc238
[Sun Nov 13 06:41:10 2022] t1 : 0000000000000001 t2 : ffffffd82339e700 s0 : ffffffd829c4bc00
[Sun Nov 13 06:41:10 2022] s1 : ffffffd82339d380 a0 : 0000000000000000 a1 : ffffffd82339d428
[Sun Nov 13 06:41:10 2022] a2 : 0000000000000000 a3 : ffffffd82339d428 a4 : ffffffd8353bc000
[Sun Nov 13 06:41:10 2022] a5 : 0000000000020000 a6 : 0000000000020000 a7 : 0000000000000003
[Sun Nov 13 06:41:10 2022] s2 : 0000000000000000 s3 : ffffffd835ab6748 s4 : 0000000000000006
[Sun Nov 13 06:41:10 2022] s5 : 0000000000000008 s6 : ffffffff03cc33d8 s7 : ffffffff03c9fda0
[Sun Nov 13 06:41:10 2022] s8 : ffffffff03c9f258 s9 : ffffffff03c9c780 s10: 0000000000000001
[Sun Nov 13 06:41:10 2022] s11: ffffffd82339d380 t3 : 0000000000000002 t4 : 0000000001700000
[Sun Nov 13 06:41:10 2022] t5 : ffffffd82339e700 t6 : 0000000000000002
[Sun Nov 13 06:41:10 2022] status: 0000000200000120 badaddr: 0000000000000060 cause: 000000000000000d
[Sun Nov 13 06:41:10 2022] [<ffffffff03c15c3e>] vdev_mirror_io_done+0x276/0x3a0 [zfs]
[Sun Nov 13 06:41:10 2022] [<ffffffff03c569d8>] zio_vdev_io_done+0xa8/0x1c6 [zfs]
[Sun Nov 13 06:41:10 2022] [<ffffffff03c564bc>] zio_execute+0xd8/0x192 [zfs]
[Sun Nov 13 06:41:10 2022] [<ffffffff0245eb98>] taskq_thread+0x272/0x41c [spl]
[Sun Nov 13 06:41:10 2022] [<ffffffff8004a87e>] kthread+0xc6/0xdc
[Sun Nov 13 06:41:10 2022] [<ffffffff80003b94>] ret_from_exception+0x0/0xc
[Sun Nov 13 06:41:10 2022] ---[ end trace 0000000000000000 ]---

[Mon Nov 14 15:30:33 2022]
Unable to handle kernel NULL pointer dereference at virtual address 0000000000000060
Oops [#1]
Modules linked in: binfmt_misc 8723ds(OE) cfg80211 zfs(POE) pwrseq_simple sunxi_cir rc_core snd_soc_hdmi_codec zunicode(POE) zzstd(OE) ses enclosure scsi_transport_sas sunxi phy_generic zlua(OE) sunxi_cedrus(C) v4l2_mem2mem videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 zcommon(POE) videobuf2_common snd_soc_simple_card sun20i_codec snd_soc_simple_card_utils snd_soc_dmic znvpair(POE) zavl(POE) videodev icp(POE) snd_soc_core leds_sun50i_r329 led_class_multicolor spl(OE) mc snd_compress ac97_bus snd_pcm_dmaengine snd_pcm snd_timer snd leds_gpio soundcore nls_iso8859_1 uio_pdrv_genirq uio ramoops pstore_blk reed_solomon efi_pstore pstore_zone dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ip_tables x_tables autofs4 efivarfs raid10 raid456 libcrc32c async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 raid0 multipath linear dw_hdmi_i2s_audio dw_hdmi_cec sun4i_drm sun8i_drm_hdmi sun4i_frontend sun8i_mixer dw_hdmi display_connector sun4i_tcon
 drm_cma_helper drm_kms_helper sun8i_tcon_top backlight syscopyarea sysfillrect sysimgblt fb_sys_fops
CPU: 0 PID: 792 Comm: z_rd_int Tainted: P C OE 5.17.0-1003-allwinner #3-Ubuntu
Hardware name: Sipeed Lichee RV Dock (DT)
epc : zio_vdev_child_io+0x34/0x104 [zfs]
 ra : vdev_mirror_io_done+0x276/0x3a0 [zfs]
epc : ffffffff04886d7a ra : ffffffff04841c3e sp : ffffffd80c3efb90
 gp : ffffffff81e93b80 tp : ffffffd80f453200 t0 : ffffffd7fffcc2c8
 t1 : 0000000000000001 t2 : ffffffd80720e220 s0 : ffffffd80c3efc00
 s1 : ffffffd80720f5a0 a0 : 0000000000000000 a1 : ffffffd80720f648
 a2 : 0000000000000000 a3 : ffffffd80720f648 a4 : ffffffd82629f680
 a5 : 0000000000020000 a6 : 0000000000020000 a7 : 0000000000000003
 s2 : 0000000000000000 s3 : ffffffd83566db08 s4 : 0000000000000006
 s5 : 0000000000000008 s6 : ffffffff048ef3d8 s7 : ffffffff048cbda0
 s8 : ffffffff048cb258 s9 : ffffffff048c8780 s10: 0000000000000001
 s11: ffffffd80720f5a0 t3 : 0000000000000002 t4 : 0000000001700000
 t5 : ffffffd80720e220 t6 : 0000000000000002
status: 0000000200000120 badaddr: 0000000000000060 cause: 000000000000000d
[<ffffffff04841c3e>] vdev_mirror_io_done+0x276/0x3a0 [zfs]
[<ffffffff048829d8>] zio_vdev_io_done+0xa8/0x1c6 [zfs]
[<ffffffff048824bc>] zio_execute+0xd8/0x192 [zfs]
[<ffffffff0228bb98>] taskq_thread+0x272/0x41c [spl]
[<ffffffff8004a87e>] kthread+0xc6/0xdc
[<ffffffff80003b94>] ret_from_exception+0x0/0xc
---[ end trace 0000000000000000 ]---

[Thu Nov 17 08:20:52 2022] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000060
[Thu Nov 17 08:20:52 2022] Oops [#1]
[Thu Nov 17 08:20:52 2022] Modules linked in: binfmt_misc 8723ds(OE) cfg80211 zfs(POE) pwrseq_simple sunxi_cir zunicode(POE) rc_core snd_soc_hdmi_codec zzstd(OE) ses enclosure scsi_transport_sas sunxi phy_generic zlua(OE) sunxi_cedrus(C) v4l2_mem2mem videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 zcommon(POE) videobuf2_common snd_soc_simple_card sun20i_codec snd_soc_dmic znvpair(POE) snd_soc_simple_card_utils zavl(POE) videodev icp(POE) leds_sun50i_r329 snd_soc_core led_class_multicolor spl(OE) mc snd_compress ac97_bus snd_pcm_dmaengine snd_pcm snd_timer snd leds_gpio soundcore nls_iso8859_1 uio_pdrv_genirq uio ramoops pstore_blk reed_solomon efi_pstore pstore_zone dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ip_tables x_tables autofs4 efivarfs raid10 raid456 libcrc32c async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 raid0 multipath linear dw_hdmi_i2s_audio dw_hdmi_cec sun4i_drm sun8i_drm_hdmi sun4i_frontend sun8i_mixer dw_hdmi sun4i_tcon display_connector
[Thu Nov 17 08:20:52 2022] drm_cma_helper drm_kms_helper backlight syscopyarea sysfillrect sun8i_tcon_top sysimgblt fb_sys_fops
[Thu Nov 17 08:20:52 2022] CPU: 0 PID: 777 Comm: z_rd_int Tainted: P C OE 5.17.0-1003-allwinner #3-Ubuntu
[Thu Nov 17 08:20:52 2022] Hardware name: Sipeed Lichee RV Dock (DT)
[Thu Nov 17 08:20:52 2022] epc : zio_vdev_child_io+0x34/0x104 [zfs]
[Thu Nov 17 08:20:52 2022] ra : vdev_mirror_io_done+0x276/0x3a0 [zfs]
[Thu Nov 17 08:20:52 2022] epc : ffffffff037efd7a ra : ffffffff037aac3e sp : ffffffd80926fb90
[Thu Nov 17 08:20:52 2022] gp : ffffffff81e93b80 tp : ffffffd80f453200 t0 : ffffffd7fffcc258
[Thu Nov 17 08:20:52 2022] t1 : 0000000000000001 t2 : ffffffd8361a5d40 s0 : ffffffd80926fc00
[Thu Nov 17 08:20:52 2022] s1 : ffffffd8361a6700 a0 : 0000000000000000 a1 : ffffffd8361a67a8
[Thu Nov 17 08:20:52 2022] a2 : 0000000000000000 a3 : ffffffd8361a67a8 a4 : ffffffd808b6e138
[Thu Nov 17 08:20:52 2022] a5 : 0000000000020000 a6 : 0000000000020000 a7 : 0000000000000003
[Thu Nov 17 08:20:52 2022] s2 : 0000000000000000 s3 : ffffffd81c936748 s4 : 0000000000000006
[Thu Nov 17 08:20:52 2022] s5 : 0000000000000008 s6 : ffffffff038583d8 s7 : ffffffff03834da0
[Thu Nov 17 08:20:52 2022] s8 : ffffffff03834258 s9 : ffffffff03831780 s10: 0000000000000001
[Thu Nov 17 08:20:52 2022] s11: ffffffd8361a6700 t3 : 0000000000000002 t4 : 0000000001700000
[Thu Nov 17 08:20:52 2022] t5 : ffffffd8361a5d40 t6 : 0000000000000002
[Thu Nov 17 08:20:52 2022] status: 0000000200000120 badaddr: 0000000000000060 cause: 000000000000000d
[Thu Nov 17 08:20:52 2022] [<ffffffff037aac3e>] vdev_mirror_io_done+0x276/0x3a0 [zfs]
[Thu Nov 17 08:20:52 2022] [<ffffffff037eb9d8>] zio_vdev_io_done+0xa8/0x1c6 [zfs]
[Thu Nov 17 08:20:52 2022] [<ffffffff037eb4bc>] zio_execute+0xd8/0x192 [zfs]
[Thu Nov 17 08:20:53 2022] [<ffffffff02285b98>] taskq_thread+0x272/0x41c [spl]
[Thu Nov 17 08:20:53 2022] [<ffffffff8004a87e>] kthread+0xc6/0xdc
[Thu Nov 17 08:20:53 2022] [<ffffffff80003b94>] ret_from_exception+0x0/0xc
[Thu Nov 17 08:20:53 2022] ---[ end trace 0000000000000000 ]---

[Fri Nov 18 04:02:02 2022] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000060
[Fri Nov 18 04:02:02 2022] Oops [#1]
[Fri Nov 18 04:02:02 2022] Modules linked in: tls binfmt_misc 8723ds(OE) cfg80211 zfs(POE) pwrseq_simple sunxi_cir zunicode(POE) rc_core snd_soc_hdmi_codec zzstd(OE) ses enclosure sunxi scsi_transport_sas phy_generic zlua(OE) sunxi_cedrus(C) v4l2_mem2mem videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common snd_soc_simple_card zcommon(POE) snd_soc_simple_card_utils sun20i_codec snd_soc_dmic znvpair(POE) zavl(POE) videodev icp(POE) leds_sun50i_r329 snd_soc_core led_class_multicolor mc spl(OE) snd_compress ac97_bus snd_pcm_dmaengine snd_pcm snd_timer snd leds_gpio soundcore nls_iso8859_1 uio_pdrv_genirq uio ramoops pstore_blk reed_solomon pstore_zone efi_pstore dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ip_tables x_tables autofs4 efivarfs raid10 raid456 libcrc32c async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 raid0 multipath linear dw_hdmi_cec dw_hdmi_i2s_audio sun8i_drm_hdmi dw_hdmi sun8i_mixer sun4i_drm sun4i_frontend sun4i_tcon sun8i_tcon_top
[Fri Nov 18 04:02:02 2022] display_connector drm_cma_helper drm_kms_helper backlight syscopyarea sysfillrect sysimgblt fb_sys_fops
[Fri Nov 18 04:02:02 2022] CPU: 0 PID: 808 Comm: z_rd_int Tainted: P C OE 5.17.0-1003-allwinner #3-Ubuntu
[Fri Nov 18 04:02:02 2022] Hardware name: Sipeed Lichee RV Dock (DT)
[Fri Nov 18 04:02:02 2022] epc : zio_vdev_child_io+0x34/0x104 [zfs]
[Fri Nov 18 04:02:02 2022] ra : vdev_mirror_io_done+0x276/0x3a0 [zfs]
[Fri Nov 18 04:02:02 2022] epc : ffffffff05336d7a ra : ffffffff052f1c3e sp : ffffffd808d27b90
[Fri Nov 18 04:02:02 2022] gp : ffffffff81e93b80 tp : ffffffd80f456400 t0 : ffffffd7fffcc2a8
[Fri Nov 18 04:02:02 2022] t1 : 0000000000000001 t2 : ffffffd82339fa80 s0 : ffffffd808d27c00
[Fri Nov 18 04:02:02 2022] s1 : ffffffd82339dd40 a0 : 0000000000000000 a1 : ffffffd82339dde8
[Fri Nov 18 04:02:02 2022] a2 : 0000000000000000 a3 : ffffffd82339dde8 a4 : ffffffd80bf72888
[Fri Nov 18 04:02:02 2022] a5 : 0000000000020000 a6 : 0000000000020000 a7 : 0000000000000003
[Fri Nov 18 04:02:02 2022] s2 : 0000000000000000 s3 : ffffffd83532c808 s4 : 0000000000000006
[Fri Nov 18 04:02:02 2022] s5 : 0000000000000008 s6 : ffffffff0539f3d8 s7 : ffffffff0537bda0
[Fri Nov 18 04:02:02 2022] s8 : ffffffff0537b258 s9 : ffffffff05378780 s10: 0000000000000001
[Fri Nov 18 04:02:02 2022] s11: ffffffd82339dd40 t3 : 0000000000000002 t4 : 0000000001700000
[Fri Nov 18 04:02:02 2022] t5 : ffffffd82339fa80 t6 : 0000000000000002
[Fri Nov 18 04:02:02 2022] status: 0000000200000120 badaddr: 0000000000000060 cause: 000000000000000d
[Fri Nov 18 04:02:02 2022] [<ffffffff052f1c3e>] vdev_mirror_io_done+0x276/0x3a0 [zfs]
[Fri Nov 18 04:02:02 2022] [<ffffffff053329d8>] zio_vdev_io_done+0xa8/0x1c6 [zfs]
[Fri Nov 18 04:02:02 2022] [<ffffffff053324bc>] zio_execute+0xd8/0x192 [zfs]
[Fri Nov 18 04:02:02 2022] [<ffffffff02255b98>] taskq_thread+0x272/0x41c [spl]
[Fri Nov 18 04:02:02 2022] [<ffffffff8004a87e>] kthread+0xc6/0xdc
[Fri Nov 18 04:02:02 2022] [<ffffffff80003b94>] ret_from_exception+0x0/0xc
[Fri Nov 18 04:02:02 2022] ---[ end trace 0000000000000000 ]---