Exporting a logical volume via NBD leads to kernel crashes

[Sebastian Unger]

I'm exporting a logical volume via nbd-server to a raspberry pi which uses the nbd device as a physical volume for its own LVM stack. Some operations such as running mkfs.ext2 on a logical volume inside the raspberry pi crash the exporting server. I have seen various messages from the kernel in the syslog & dmesg and experienced lock-ups etc. The operations that cause this cause the issues consistently. For instance the mkfs.ext2 on the RPI cause this to appear in dmesg on the server exporting the NBD:

[ 1188.468279] ------------[ cut here ]------------
[ 1188.468282] kernel BUG at include/linux/highmem.h:279!
[ 1188.468289] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[ 1188.468292] CPU: 2 PID: 4944 Comm: pool Not tainted 5.19.0-32-generic #33~22.04.1-Ubuntu
[ 1188.468295] Hardware name: Gigabyte Technology Co., Ltd. X570 GAMING X/X570 GAMING X, BIOS F37 12/26/2022
[ 1188.468296] RIP: 0010:zero_user_segments.constprop.0+0x15c/0x1a0
[ 1188.468303] Code: ff 48 8b 57 48 f6 c2 01 0f 84 00 ff ff ff 48 83 ea 01 48 39 d7 0f 84 f3 fe ff ff 41 be 00 10 00 00 4c 39 f0 0f 86 03 ff ff ff <0f> 0b f7 c3 ff 0f 00 00 0f 85 04 ff ff ff 48 8b 03 a9 00 00 01 00
[ 1188.468305] RSP: 0018:ffffbe5984817be8 EFLAGS: 00010206
[ 1188.468308] RAX: 00000000fae00000 RBX: fffff46887f2f800 RCX: 0000000000000000
[ 1188.468309] RDX: 0017ffffc0020037 RSI: 0000000000000000 RDI: fffff46887f2f800
[ 1188.468311] RBP: ffffbe5984817c20 R08: fffff46880000000 R09: 0000000000000000
[ 1188.468312] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000fae00000
[ 1188.468314] R13: 0000000000000000 R14: 0000000000001000 R15: ffff977ac0000000
[ 1188.468315] FS: 00007fd36bfff640(0000) GS:ffff9782dea80000(0000) knlGS:0000000000000000
[ 1188.468317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1188.468319] CR2: 000004c500e89000 CR3: 0000000114472000 CR4: 0000000000350ee0
[ 1188.468321] Call Trace:
[ 1188.468323] <TASK>
[ 1188.468325] truncate_inode_partial_folio+0xce/0x240
[ 1188.468329] truncate_inode_pages_range+0x237/0x650
[ 1188.468332] ? hrtimer_cancel+0x21/0x50
[ 1188.468335] ? futex_wait+0x238/0x260
[ 1188.468339] ? __hrtimer_init+0x120/0x120
[ 1188.468342] ? rseq_get_rseq_cs.isra.0+0x1b/0x270
[ 1188.468345] ? bd_prepare_to_claim+0x131/0x150
[ 1188.468349] ? rseq_ip_fixup+0x72/0x1b0
[ 1188.468351] truncate_bdev_range+0x4f/0x120
[ 1188.468354] blkdev_common_ioctl+0x479/0x980
[ 1188.468358] blkdev_ioctl+0x133/0x2a0
[ 1188.468360] __x64_sys_ioctl+0x9a/0xe0
[ 1188.468363] do_syscall_64+0x59/0x90
[ 1188.468367] ? syscall_exit_to_user_mode+0x2a/0x50
[ 1188.468370] ? do_syscall_64+0x69/0x90
[ 1188.468372] ? do_syscall_64+0x69/0x90
[ 1188.468375] ? syscall_exit_to_user_mode+0x2a/0x50
[ 1188.468377] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 1188.468380] RIP: 0033:0x7fd371d1aaff
[ 1188.468383] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <41> 89 c0 3d 00 f0 ff ff 77 1f 48 8b 44 24 18 64 48 2b 04 25 28 00
[ 1188.468385] RSP: 002b:00007fd36bffeb00 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 1188.468387] RAX: ffffffffffffffda RBX: 00007fd36bffebb0 RCX: 00007fd371d1aaff
[ 1188.468389] RDX: 00007fd36bffebb0 RSI: 0000000000001277 RDI: 0000000000000007
[ 1188.468390] RBP: 00007fd36bffec00 R08: 0000000000000000 R09: 0000000000000001
[ 1188.468391] R10: fffffffbfae00000 R11: 0000000000000246 R12: 0000557743114ea0
[ 1188.468393] R13: 0000557743114e40 R14: 00005577431117d0 R15: 0000000000000007
[ 1188.468395] </TASK>
[ 1188.468396] Modules linked in: rfcomm snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi bridge stp snd_hda_intel llc snd_intel_dspcfg cmac snd_intel_sdw_acpi algif_hash snd_hda_codec algif_skcipher snd_hda_core af_alg snd_hwdep intel_rapl_msr snd_pcm intel_rapl_common bnep snd_seq_midi btusb snd_seq_midi_event btrtl snd_rawmidi edac_mce_amd btbcm sch_fq_codel snd_seq kvm_amd btintel btmtk msr snd_seq_device binfmt_misc kvm bluetooth snd_timer parport_pc ftdi_sio nls_iso8859_1 snd ecdh_generic rapl ppdev serio_raw gigabyte_wmi wmi_bmof usbserial joydev input_leds k10temp ccp ecc soundcore lp ramoops mac_hid pstore_blk parport reed_solomon pstore_zone efi_pstore ip_tables x_tables autofs4 amdgpu iommu_v2 gpu_sched i2c_algo_bit dm_thin_pool drm_ttm_helper hid_generic dm_persistent_data ttm usbhid dm_bio_prison drm_display_helper hid dm_bufio cec libcrc32c rc_core drm_kms_helper syscopyarea sysfillrect crct10dif_pclmul sysimgblt crc32_pclmul ghash_clmulni_intel
[ 1188.468452] aesni_intel crypto_simd fb_sys_fops nvme ahci xhci_pci r8169 cryptd psmouse drm i2c_piix4 nvme_core libahci xhci_pci_renesas realtek wmi
[ 1188.468463] ---[ end trace 0000000000000000 ]---
[ 1188.614812] RIP: 0010:zero_user_segments.constprop.0+0x15c/0x1a0
[ 1188.614818] Code: ff 48 8b 57 48 f6 c2 01 0f 84 00 ff ff ff 48 83 ea 01 48 39 d7 0f 84 f3 fe ff ff 41 be 00 10 00 00 4c 39 f0 0f 86 03 ff ff ff <0f> 0b f7 c3 ff 0f 00 00 0f 85 04 ff ff ff 48 8b 03 a9 00 00 01 00
[ 1188.614820] RSP: 0018:ffffbe5984817be8 EFLAGS: 00010206
[ 1188.614823] RAX: 00000000fae00000 RBX: fffff46887f2f800 RCX: 0000000000000000
[ 1188.614825] RDX: 0017ffffc0020037 RSI: 0000000000000000 RDI: fffff46887f2f800
[ 1188.614827] RBP: ffffbe5984817c20 R08: fffff46880000000 R09: 0000000000000000
[ 1188.614829] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000fae00000
[ 1188.614831] R13: 0000000000000000 R14: 0000000000001000 R15: ffff977ac0000000
[ 1188.614833] FS: 00007fd36bfff640(0000) GS:ffff9782dea80000(0000) knlGS:0000000000000000
[ 1188.614835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1188.614838] CR2: 000004c500e89000 CR3: 0000000114472000 CR4: 0000000000350ee0

The server-side setup uses the package nbd-server, a udev-rule to make certain server-side LVs accessible to ndb-server and a single export config file:

sudo apt install nbd-server
sudo install -m 0644 /dev/fd/0 /etc/udev/rules.d/99-nbd.rules <<<'ENV{DM_VG_NAME}=="vg0" ENV{DM_LV_NAME}=="nbd-*" OWNER="nbd"'
sudo install -m 0644 -D /dev/fd/0 /etc/nbd-server/conf.d/ws4051.conf <<-'EOF'
[ws4051]
splice=true
exportname=/dev/vg0/nbd-ws4051
trim=true
flush=true
fua=true
EOF
sudo lvcreate -n nbd-ws4051 -L 32G vg0

The client (raspberry pi) uses packages linux-modules-extra-raspi and nbd-client and adds the following to the kernel command-line (192.168.5.20 is the nbd servers IP):
net.ifnames=0 ip=:::::eth0:dhcp nbdroot=192.168.5.20,ws4051,nbd0

The block device so imported is a LVM PV which contains the PIs root FS etc as well as a LV boot. It is this LV that I am running mkfs.ext2 on.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: linux-image-5.19.0-32-generic 5.19.0-32.33~22.04.1
ProcVersionSignature: Ubuntu 5.19.0-32.33~22.04.1-generic 5.19.17
Uname: Linux 5.19.0-32-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.3
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Mon Feb 20 10:18:47 2023
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_NZ.UTF-8
 SHELL=/bin/bash
SourcePackage: linux-signed-hwe-5.19
UpgradeStatus: No upgrade log present (probably fresh install)