dm thin block allocation failure

[Markus Schade]

When using the bionic kernel (hwe-edge) on xenial we randomly trigger a kernel bug when creating a thin provisioned lvm volume.

kernel: [146487.421278] ------------[ cut here ]------------
kernel: [146487.421281] kernel BUG at /build/linux-hwe-edge-eBL7So/linux-hwe-edge-4.15.0/drivers/md/persistent-data/dm-space-map-disk.c:178!
kernel: [146487.421435] invalid opcode: 0000 [#1] SMP PTI
kernel: [146487.421528] Modules linked in: dm_snapshot cmac arc4 md4 nls_utf8 cifs ccm fscache dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM xt_nat iptable_nat nf_nat_ipv4 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack iptable_filter veth dummy bridge iptable_mangle xt_CT iptable_raw ip_tables x_tables vhost_net vhost tap ipmi_ssif dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio intel_rapl skx_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf lpc_ich ioatdma shpchp dca acpi_power_meter acpi_pad mac_hid ipmi_si ipmi_devintf ipmi_msghandler
kernel: [146487.421859] nf_nat_ftp nf_conntrack_ftp nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_proto_gre nf_nat nf_conntrack_proto_gre nf_conntrack tcp_htcp 8021q garp mrp stp llc bonding autofs4 btrfs zstd_compress raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear raid10 raid1 i2c_algo_bit ttm i40e drm_kms_helper syscopyarea ptp sysfillrect pps_core sysimgblt fb_sys_fops nvme ahci drm nvme_core libahci wmi
kernel: [146487.422080] CPU: 13 PID: 2274450 Comm: kworker/u576:1 Not tainted 4.15.0-22-generic #24~16.04.1-Ubuntu
kernel: [146487.422211] Hardware name: Quanta Cloud Technology Inc. QuantaGrid D52B-1U/S5B-MB (LBG-4), BIOS 3A10.03 11/17/2017
kernel: [146487.422349] Workqueue: dm-thin do_worker [dm_thin_pool]
kernel: [146487.422443] RIP: 0010:sm_disk_new_block+0xa1/0xb0 [dm_persistent_data]
kernel: [146487.422536] RSP: 0018:ffff9efc5e4d3c68 EFLAGS: 00010297
kernel: [146487.422626] RAX: 0000000000000000 RBX: ffff88b122d34000 RCX: 0000000000000000
kernel: [146487.422749] RDX: ffff8868d8631e00 RSI: 0000000000000282 RDI: ffff88b137691200
kernel: [146487.422872] RBP: ffff9efc5e4d3c90 R08: 0000000000000000 R09: ffff9efc5e4d39f0
kernel: [146487.422994] R10: ffff9efc5e4d38f8 R11: ffff9efc5e4d3b48 R12: 0000000000000000
kernel: [146487.423117] R13: ffff9efc5e4d3d40 R14: ffff9efc5e4d3d40 R15: ffff882de1769fa0
kernel: [146487.423240] FS: 0000000000000000(0000) GS:ffff8868ee540000(0000) knlGS:0000000000000000
kernel: [146487.423366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [146487.423457] CR2: 00007f0f30b083c0 CR3: 0000002ab260a004 CR4: 00000000007626e0
kernel: [146487.423580] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: [146487.423703] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
kernel: [146487.423826] PKRU: 55555554
kernel: [146487.423906] Call Trace:
kernel: [146487.423989] dm_pool_alloc_data_block+0x44/0x60 [dm_thin_pool]
kernel: [146487.424082] alloc_data_block.isra.53+0x6f/0x180 [dm_thin_pool]
kernel: [146487.424175] process_cell+0x2cf/0x4d0 [dm_thin_pool]
kernel: [146487.424267] ? mempool_alloc_slab+0x15/0x20
kernel: [146487.424354] ? mempool_alloc+0x73/0x180
kernel: [146487.424440] ? process_prepared+0x8d/0xc0 [dm_thin_pool]
kernel: [146487.424531] do_worker+0x25c/0x8b0 [dm_thin_pool]
kernel: [146487.424621] process_one_work+0x14d/0x410
kernel: [146487.424708] ? process_discard_cell_passdown+0x1e0/0x1e0 [dm_thin_pool]
kernel: [146487.424802] ? process_one_work+0x14d/0x410
kernel: [146487.424888] worker_thread+0x4b/0x460
kernel: [146487.424973] kthread+0x105/0x140
kernel: [146487.425055] ? process_one_work+0x410/0x410
kernel: [146487.425142] ? kthread_associate_blkcg+0xa0/0xa0
kernel: [146487.425230] ? do_syscall_64+0x73/0x130
kernel: [146487.425315] ? SyS_exit_group+0x14/0x20
kernel: [146487.425401] ret_from_fork+0x35/0x40
kernel: [146487.425485] Code: c0 01 48 89 83 20 22 00 00 49 8b 75 00 e8 68 f7 ff ff 85 c0 75 10 83 7d dc 01 75 0f 48 83 83 28 22 00 00 01 eb af 41 89 c4 eb aa <0f> 0b e8 e8 1b cf d2 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55
kernel: [146487.425703] RIP: sm_disk_new_block+0xa1/0xb0 [dm_persistent_data] RSP: ffff9efc5e4d3c68
kernel: [146487.425869] ---[ end trace 76985d9f3818cc3d ]---

[Markus Schade (lp-markusschade)]

Also present in HWE kernel 4.15.0-23-generic #25~16.04.1-Ubuntu for xenial

kernel: [450194.803852] ------------[ cut here ]------------
kernel: [450194.803855] kernel BUG at /build/linux-hwe-edge-ew4Cb4/linux-hwe-edge-4.15.0/drivers/md/persistent-data/dm-space-map-disk.c:178!
kernel: [450194.803986] invalid opcode: 0000 [#1] SMP PTI
kernel: [450194.804101] Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm fscache dm_snapshot dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM xt_nat iptable_nat nf_nat_ipv4 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_CT iptable_raw xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack iptable_filter veth dummy bridge iptable_mangle ip_tables x_tables vhost_net vhost tap intel_rapl skx_edac x86_pkg_temp_thermal intel_powerclamp coretemp dm_thin_pool dm_persistent_data dm_bio_prison kvm_intel dm_bufio kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc ipmi_ssif aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf lpc_ich shpchp ioatdma dca acpi_power_meter acpi_pad mac_hid ipmi_si ipmi_devintf ipmi_msghandler
kernel: [450194.804489] nf_nat_ftp nf_conntrack_ftp nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_proto_gre nf_nat nf_conntrack_proto_gre nf_conntrack tcp_htcp 8021q garp mrp stp llc bonding autofs4 btrfs zstd_compress raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear raid1 raid10 i2c_algo_bit ttm i40e drm_kms_helper ptp syscopyarea pps_core sysfillrect nvme sysimgblt fb_sys_fops nvme_core drm ahci libahci wmi
kernel: [450194.804763] CPU: 4 PID: 1788072 Comm: kworker/u128:1 Not tainted 4.15.0-23-generic #25~16.04.1-Ubuntu
kernel: [450194.804898] Hardware name: Quanta Cloud Technology Inc. QuantaGrid D52B-1U/S5B-MB (LBG-4), BIOS 3A10.03 11/17/2017
kernel: [450194.805037] Workqueue: dm-thin do_worker [dm_thin_pool]
kernel: [450194.805132] RIP: 0010:sm_disk_new_block+0xa1/0xb0 [dm_persistent_data]
kernel: [450194.805225] RSP: 0018:ffffbd5c00d3fc68 EFLAGS: 00010297
kernel: [450194.805315] RAX: 0000000000000000 RBX: ffff98ff57aa0000 RCX: 0000000000000000
kernel: [450194.805437] RDX: ffff9948019ebc00 RSI: 0000000000000282 RDI: ffff98ff99bf4a00
kernel: [450194.805561] RBP: ffffbd5c00d3fc90 R08: 0000000000000000 R09: ffffbd5c00d3f9f0
kernel: [450194.805685] R10: ffffbd5c00d3f8f8 R11: ffffbd5c00d3fb48 R12: 0000000000000000
kernel: [450194.805808] R13: ffffbd5c00d3fd40 R14: ffffbd5c00d3fd40 R15: ffff98dc797fcce0
kernel: [450194.805934] FS: 0000000000000000(0000) GS:ffff98ffaff00000(0000) knlGS:0000000000000000
kernel: [450194.806061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [450194.806152] CR2: 000000c420224000 CR3: 000000544820a003 CR4: 00000000007626e0
kernel: [450194.806275] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: [450194.806399] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
kernel: [450194.806522] PKRU: 55555554
kernel: [450194.806604] Call Trace:
kernel: [450194.806690] dm_pool_alloc_data_block+0x44/0x60 [dm_thin_pool]
kernel: [450194.806783] al...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1777398

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.17 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.18-rc1

[Markus Schade (lp-markusschade)]

Hi Joseph,

sure, but running an rc- kernel is not ideal for production systems, so I can only give this limited exposure on a few canary systems.
I hope we can find a way to deterministically trigger this, to confirm a fix.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Markus Schade (lp-markusschade)]

This issue is still present. Possibly fixed by upstream commit
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/md?id=a685557fbbc3122ed11e8ad3fa63a11ebc5de8c3

https://www.redhat.com/archives/dm-devel/2018-April/msg00022.html

kernel: [8860597.921484] kernel BUG at /build/linux-hwe-okqLve/linux-hwe-4.15.0/drivers/md/persistent-data/dm-space-map-disk.c:178!
kernel: [8860597.921579] invalid opcode: 0000 [#1] SMP PTI
kernel: [8860597.921640] Modules linked in: cpuid cmac arc4 md4 nls_utf8 cifs ccm fscache dm_snapshot dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM xt_nat iptable_nat nf_nat_ipv4 iptable_mangle xt_CT iptable_raw veth dummy nf_conntrack_ipv4 nf_defrag_ipv4 bridge iptable_filter ip_tables xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip6table_filter ip6_tables x_tables vhost_net vhost tap intel_rapl skx_edac x86_pkg_temp_thermal intel_powerclamp coretemp dm_thin_pool kvm_intel dm_persistent_data dm_bio_prison kvm dm_bufio irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd ipmi_ssif glue_helper cryptd intel_cstate intel_rapl_perf lpc_ich shpchp ioatdma dca acpi_pad acpi_power_meter mac_hid ipmi_si ipmi_devintf
kernel: [8860597.921973] ipmi_msghandler nf_nat_ftp nf_conntrack_ftp nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_proto_gre nf_nat nf_conntrack_proto_gre nf_conntrack tcp_htcp 8021q garp mrp stp llc bonding autofs4 btrfs zstd_compress raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear raid1 raid10 i40e i2c_algo_bit ttm ptp drm_kms_helper pps_core syscopyarea nvme sysfillrect sysimgblt fb_sys_fops nvme_core drm ahci libahci wmi
kernel: [8860597.922282] CPU: 38 PID: 1917945 Comm: kworker/u128:2 Not tainted 4.15.0-31-generic #33~16.04.1-Ubuntu
kernel: [8860597.922413] Hardware name: Quanta Cloud Technology Inc. QuantaGrid D52B-1U/S5B-MB (LBG-4), BIOS 3A10.10 02/12/2018
kernel: [8860597.922554] Workqueue: dm-thin do_worker [dm_thin_pool]
kernel: [8860597.922650] RIP: 0010:sm_disk_new_block+0xa1/0xb0 [dm_persistent_data]
kernel: [8860597.922747] RSP: 0018:ffffbfa19dc7fc68 EFLAGS: 00010297
kernel: [8860597.922839] RAX: 0000000000000000 RBX: ffff9e6c63f0c000 RCX: 0000000000000000
kernel: [8860597.922962] RDX: ffff9e6c3768d940 RSI: 0000000000000282 RDI: ffff9eb4dc4f3400
kernel: [8860597.923086] RBP: ffffbfa19dc7fc90 R08: 0000000000000000 R09: ffffbfa19dc7f9f0
kernel: [8860597.923211] R10: ffffbfa19dc7f8f8 R11: ffffbfa19dc7fb48 R12: 0000000000000000
kernel: [8860597.923335] R13: ffffbfa19dc7fd40 R14: ffffbfa19dc7fd40 R15: ffff9e47cb3460e0
kernel: [8860597.923460] FS: 0000000000000000(0000) GS:ffff9e6cb0380000(0000) knlGS:0000000000000000
kernel: [8860597.923587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [8860597.923678] CR2: 00000000009cfd20 CR3: 0000008d2400a003 CR4: 00000000007626e0
kernel: [8860597.923802] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: [8860597.923926] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
kernel: [8860597.924049] PKRU: 55555554
...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1777398

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Markus Schade (lp-markusschade)]

Still present in 4.15.0-47-generic even though a number of dm/thin fixes have been backported

kernel: [1120717.622665] Modules linked in: cpuid xt_CHECKSUM xt_nat iptable_nat nf_nat_ipv4 dm_snapshot veth dummy bridge nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_CT iptable_raw iptable_mangle xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack iptable_filter ip_tables x_tables vhost_net vhost tap dm_thin_pool dm_persistent_data intel_rapl dm_bio_prison skx_edac dm_bufio x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc ipmi_ssif aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf lpc_ich shpchp ioatdma dca acpi_power_meter acpi_pad mac_hid ipmi_si ipmi_devintf ipmi_msghandler nf_nat_ftp nf_conntrack_ftp nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_proto_gre
kernel: [1120717.623364] nf_nat nf_conntrack_proto_gre nf_conntrack tcp_htcp 8021q garp mrp stp llc bonding autofs4 btrfs zstd_compress raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear raid1 raid10 i2c_algo_bit ttm drm_kms_helper i40e syscopyarea sysfillrect sysimgblt fb_sys_fops ptp ahci nvme pps_core drm nvme_core libahci wmi
kernel: [1120717.623739] CPU: 40 PID: 1269520 Comm: kworker/u128:2 Not tainted 4.15.0-47-generic #50~16.04.1-Ubuntu
kernel: [1120717.623881] Hardware name: Quanta Cloud Technology Inc. QuantaGrid D52B-1U/S5B-MB (LBG-2), BIOS 3A13.Q402 11/16/2018
kernel: [1120717.624035] Workqueue: dm-thin do_worker [dm_thin_pool]
kernel: [1120717.624142] RIP: 0010:sm_disk_new_block+0xa1/0xb0 [dm_persistent_data]
kernel: [1120717.624245] RSP: 0018:ffffa467364d7c68 EFLAGS: 00010297
kernel: [1120717.624347] RAX: 0000000000000000 RBX: ffff89285e3f8000 RCX: 0000000000000000
kernel: [1120717.624481] RDX: ffff890b51e68000 RSI: 0000000000000282 RDI: ffff8970be770600
kernel: [1120717.624614] RBP: ffffa467364d7c90 R08: 0000000000000000 R09: ffffa467364d79f0
kernel: [1120717.624747] R10: ffffa467364d78f8 R11: ffffa467364d7b48 R12: 0000000000000000
kernel: [1120717.624880] R13: ffffa467364d7d40 R14: ffffa467364d7d40 R15: ffff8921d2ac9460
kernel: [1120717.625015] FS: 0000000000000000(0000) GS:ffff892870400000(0000) knlGS:0000000000000000
kernel: [1120717.625149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [1120717.625249] CR2: 00007f864a646000 CR3: 000000806a40a002 CR4: 00000000007626e0
kernel: [1120717.625384] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: [1120717.625517] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
kernel: [1120717.625649] PKRU: 55555554
kernel: [1120717.625739] Call Trace:
kernel: [1120717.625839] dm_pool_alloc_data_block+0x44/0x60 [dm_thin_pool]
kernel: [1120717.625948] alloc_data_block.isra.56+0x6f/0x180 [dm_thin_pool]
kernel: [1120717.626053] process_cell+0x2cf/0x4d0 [dm_thin_pool]
kernel: [1120717.626157] ? sort+0x165/0x250
kernel: [1120717.626250] do_worker+0x25c/0x8b0 [dm_thin_pool]
kernel: [1120717.626351] process_one_work+0x14d/0x410
kernel: [1120717.626451] ? process_discard_cell_passdown...

[Markus Schade (lp-markusschade)]

Currently testing 5.0.10 as we think the issue is related to thin trying to reallocate a discarded block not working correctly

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/md?id=cffd425b90147296c627dfb13f7ea8f361d962b2

[Markus Schade (lp-markusschade)]

Unfortunately still present in mainline 5.0.13 kernel:

------------[ cut here ]------------
kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178!
invalid opcode: 0000 [#1] SMP PTI
CPU: 5 PID: 1675751 Comm: kworker/u128:1 Not tainted 5.0.13-050013-generic #201905051330
Hardware name: Quanta Cloud Technology Inc. QuantaGrid D52B-1U/S5B-MB (LBG-2), BIOS 3A13.Q402 11/16/2018
Workqueue: dm-thin do_worker [dm_thin_pool]
RIP: 0010:sm_disk_new_block+0x92/0xa0 [dm_persistent_data]
Code: 55 e4 48 83 c0 01 48 89 83 20 22 00 00 49 8b 34 24 e8 82 f7 ff ff 85 c0 75 c4 83 7d e4 01 75 0a 48 83 83 28 22 00 00 01 eb b4 <0f> 0b e8 97 b4 dd c7 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55 48 89
RSP: 0018:ffffb647bb06fca8 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffff95dc1d9a4000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff95dc347a9a00
RBP: ffffb647bb06fcc8 R08: 0000000000000000 R09: ffffb647bb06fa38
R10: fffff6c3d4ca9080 R11: 0000000000000001 R12: ffffb647bb06fd70
R13: ffffb647bb06fd70 R14: ffff9593a3d6f000 R15: ffff957adcb455b0
FS: 0000000000000000(0000) GS:ffff9593efd40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c4202d2318 CR3: 00000079b7c0e002 CR4: 00000000007626e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
 dm_pool_alloc_data_block+0x3c/0x60 [dm_thin_pool]
 alloc_data_block.isra.61+0x6c/0x1a0 [dm_thin_pool]
 process_cell+0x2cd/0x4f0 [dm_thin_pool]
 ? _cond_resched+0x19/0x30
 ? u32_swap+0x10/0x10
 ? sort+0x13b/0x220
 ? u32_swap+0x10/0x10
 do_worker+0x27c/0x900 [dm_thin_pool]
 ? __switch_to_asm+0x40/0x70
 process_one_work+0x20f/0x410
 worker_thread+0x34/0x400
 kthread+0x120/0x140
 ? process_one_work+0x410/0x410
 ? __kthread_parkme+0x70/0x70
 ret_from_fork+0x35/0x40
Modules linked in: dm_snapshot xt_CHECKSUM xt_nat iptable_nat nf_nat_ipv4 veth dummy bridge ip6table_filter ip6_tables iptable_mangle xt_CT iptable_raw xt_tcpudp xt_conntrack iptable_filter ip_tables x_tables bpfilter vhost_net vhost tap dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio intel_rapl skx_edac nfit x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_cstate ioatdma ipmi_ssif intel_rapl_perf lpc_ich dca acpi_power_meter acpi_pad mac_hid ipmi_si ipmi_devintf ipmi_msghandler nf_nat_ftp nf_conntrack_ftp nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat nf_conntrack_proto_gre nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 tcp_htcp 8021q garp mrp stp llc bonding autofs4 btrfs zstd_compress raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear raid10 raid1 i2c_algo_bit ttm
 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm i40e nvme ahci nvme_core libahci wmi
---[ end trace 678c665ac774d0d4 ]---
RIP: 0010:sm_disk_new_block+0x92/0xa0 [dm_persistent_data]
Code: 55 e4 48 83 c0 01 48 89 83 20 22 00 00 49 8b 34 24 e8 82 f7 ff ff 85 c0 75 c4 83 7d e4 01 75 0a 48 83 83 28 22 00 00 01 ...

[Eric Wheeler (ewheelerinc)]

We see this in 4.19.y when using scsi_mod.use_blk_mq=y . Marcus, are you using blk-mq?

Jump in on this thread, or I can cc you if you send me an email:

https://<email address hidden>/

-Eric

[Markus Schade (lp-markusschade)]

Patch posted at
https://www.redhat.com/archives/dm-devel/2020-January/msg00021.html

[Markus Schade (lp-markusschade)]

The patch is merged upstream and will be in 5.6.

https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.6&id=4feaef830de7ffdd8352e1fe14ad3bf13c9688f8
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/md?id=e9f8ca0ae7b7bc9a032b429929431c626a69dd5e

Since we had also peformance regressions with bionics 4.15 due to
https://<email address hidden>/T/
we have moved to the eoan hwe kernel.

We have been running that with this patch so far without triggering the condition again.

[Markus Schade (lp-markusschade)]

Patch has been backported to all stable kernel releases, e.g. 4.19:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-4.19.y&id=1fac9f574cd2536abcdd645ee23d35c0e819d7ec

[Markus Schade (lp-markusschade)]

fix was included in 4.4.0-177.207 (xenial), 4.15.0-92.93 (bionic), 5.3.0-46.38 (eoan), 5.4.0-15.18 (focal)