[VROC] Kernel panic during reshape - probably missing patch

[Kinga Tanska]

Bug found for Jammy (22.04)

Steps to reproduce:
1. Create 3 partitions with parted tool on initial non-RAID drive
# parted -s /dev/nvme6n1 mklabel gpt
# parted -s /dev/nvme6n1 mkpart primary ext4 0% 30%
# parted -s /dev/nvme6n1 mkpart primary btrfs 30% 60%
# parted -s /dev/nvme6n1 mkpart primary xfs 60% 90%

2. Create filesystem on each partition
# mkfs.ext4 /dev/nvme6n1p1
# mkfs.ext4 /dev/nvme6n1p2
# mkfs.ext4 /dev/nvme6n1p3

4. Create RAID container
# mdadm --create /dev/md/imsm0 --metadata=imsm --raid-devices=4 /dev/nvme6n1 /dev/nvme4n1 /dev/nvme2n1 /dev/nvme1n1 --run –force

5. Create RAID on drive with data
# mdadm --create /dev/md/volume --level=0 --chunk 64 --raid-devices=1 /dev/nvme6n1 --run –force

6. Make sure GPT data structures are in proper place
# sgdisk -e /dev/md/volume

7. Grow temporary RAID 0 volume with two devices. Wait for reshape end.
# mdadm --grow /dev/md/imsm0 --raid-devices=2

8. Grow temporary RAID 0 to RAID 10
# mdadm --grow /dev/md/volume --level=10

Results:
Recovery starts but frozen. Kernel BUG occurs:
[ 724.885252] md: recovery of RAID array md126

[ 724.885956] ------------[ cut here ]------------

[ 724.885960] kernel BUG at drivers/md/raid10.c:928!

[ 724.891527] invalid opcode: 0000 [#1] SMP NOPTI

[ 724.896736] CPU: 5 PID: 4872 Comm: md126_resync Not tainted 5.15.0-25-generic #25-Ubuntu

[ 724.905943] Hardware name: Intel Corporation ArcherCity/ArcherCity, BIOS EGSDCRB1.SYS.0079.D34.2205030421 05/03/2022

[ 724.917839] RIP: 0010:raise_barrier+0x184/0x190 [raid10]

[ 724.923896] Code: 89 f7 e8 4f c6 cf d4 e9 e5 fe ff ff 8b 8f ec 00 00 00 85 c9 74 14 4c 8d a7 dc 00 00 00 4c 89 e7 e8 91 7f 93 d5 e9 c7 fe ff ff <0f> 0b e8 35 9b 92 d5 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 41

[ 724.945031] RSP: 0018:ff3c62714464fbd8 EFLAGS: 00010246

[ 724.950978] RAX: 0000000000000000 RBX: ff3282bd578e4c00 RCX: 0000000000000000

[ 724.959064] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ff3282bd578e4c00

[ 724.967154] RBP: ff3c62714464fc28 R08: ff3282b9cf23ca00 R09: 0000000000000001

[ 724.975239] R10: 0000000000000293 R11: ff3282bcae577590 R12: ff3282b9cf23ca00

[ 724.983306] R13: 0000000000000003 R14: 0000000000000080 R15: ff3282bd578e4c00

[ 724.991405] FS: 0000000000000000(0000) GS:ff3282bcae540000(0000) knlGS:0000000000000000

[ 725.000572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

[ 725.007105] CR2: 00007f086b758a50 CR3: 0000000044010001 CR4: 0000000000771ee0

[ 725.015209] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

[ 725.023295] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400

[ 725.031391] PKRU: 55555554

[ 725.034514] Call Trace:

[ 725.037360] <TASK>

[ 725.039787] ? bio_reset+0x12/0x50

[ 725.043721] ? raid10_alloc_init_r10buf+0x6d/0xc0 [raid10]

[ 725.049966] raid10_sync_request+0x77c/0x1600 [raid10]

[ 725.055832] md_do_sync.cold+0x42c/0xa17

[ 725.060331] md_thread+0xaa/0x160

[ 725.064144] ? md_write_inc+0x50/0x50

[ 725.068350] kthread+0x127/0x150

[ 725.072055] ? set_kthread_struct+0x50/0x50

[ 725.076850] ret_from_fork+0x1f/0x30

[ 725.080963] </TASK>

[ 725.083515] Modules linked in: xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_counter bridge stp llc nf_tables nfnetlink nvme_fabrics intel_rapl_msr intel_rapl_common i10nm_edac nfit x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_realtek snd_hda_codec_generic kvm_intel ledtrig_audio kvm snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec rapl snd_hda_core snd_hwdep snd_pcm snd_timer snd efi_pstore joydev input_leds isst_if_mbox_pci pmt_telemetry idxd pmt_crashlog isst_if_mmio mei_me soundcore pmt_class isst_if_common mei idxd_bus ipmi_ssif acpi_ipmi ipmi_si acpi_power_meter acpi_pad mac_hid sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ipmi_devintf ipmi_msghandler msr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor

[ 725.083585] async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror dm_region_hash dm_log nvme nvme_core hid_generic usbhid ast i2c_algo_bit hid drm_vram_helper drm_ttm_helper ttm ax88179_178a drm_kms_helper usbnet syscopyarea uas mii sysfillrect usb_storage sysimgblt crct10dif_pclmul fb_sys_fops crc32_pclmul ghash_clmulni_intel cec rc_core aesni_intel i2c_i801 crypto_simd vmd i2c_smbus intel_pmt i2c_ismt drm cryptd xhci_pci ahci igc libahci xhci_pci_renesas wmi pinctrl_emmitsburg

[ 725.230991] ---[ end trace ba2ba247d7919b18 ]---

[ 725.482652] RIP: 0010:raise_barrier+0x184/0x190 [raid10]

[ 725.489216] Code: 89 f7 e8 4f c6 cf d4 e9 e5 fe ff ff 8b 8f ec 00 00 00 85 c9 74 14 4c 8d a7 dc 00 00 00 4c 89 e7 e8 91 7f 93 d5 e9 c7 fe ff ff <0f> 0b e8 35 9b 92 d5 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 41

[ 725.511334] RSP: 0018:ff3c62714464fbd8 EFLAGS: 00010246

[ 725.517805] RAX: 0000000000000000 RBX: ff3282bd578e4c00 RCX: 0000000000000000

[ 725.526397] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ff3282bd578e4c00

[ 725.535005] RBP: ff3c62714464fc28 R08: ff3282b9cf23ca00 R09: 0000000000000001

[ 725.543589] R10: 0000000000000293 R11: ff3282bcae577590 R12: ff3282b9cf23ca00

[ 725.552157] R13: 0000000000000003 R14: 0000000000000080 R15: ff3282bd578e4c00

[ 725.560768] FS: 0000000000000000(0000) GS:ff3282bcae540000(0000) knlGS:0000000000000000

[ 725.570413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

[ 725.577411] CR2: 00007f086b758a50 CR3: 0000000044010001 CR4: 0000000000771ee0

[ 725.585976] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

[ 725.594531] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400

[ 725.603077] PKRU: 55555554

Probably commit is missed:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=0f2571ad7a30ff6b33cde142439f9378669f8b4f

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1984117

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Kinga Tanska (ktanska)]

Logs are added in description instead of attachment. Changing to confirmed.

[Kinga Tanska (ktanska)]

Hello,

do you have any update on this issue?

Thanks,
Kinga Tanska

[Kinga Tanska (ktanska)]

Due to closing our project in Intel, there is no reason to process it.
Please close this issue.