| 2023-10-14 21:20:59 |
Thomas Debesse |
bug |
|
|
added bug |
| 2023-10-14 21:30:09 |
Ubuntu Kernel Bot |
linux (Ubuntu): status |
New |
Confirmed |
|
| 2023-10-16 11:52:39 |
Juerg Haefliger |
tags |
amd64 apport-bug mantic |
amd64 apport-bug kernel-flexible-array mantic |
|
| 2023-10-16 13:04:03 |
Matthew Mirvish |
bug |
|
|
added subscriber Matthew Mirvish |
| 2023-10-24 15:08:22 |
Jonathan Crooke |
attachment added |
|
log.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2039368/+attachment/5712911/+files/log.txt |
|
| 2023-10-24 15:08:34 |
Jonathan Crooke |
bug |
|
|
added subscriber Jonathan Crooke |
| 2023-11-22 15:51:54 |
Eduardo-sanchez-mata |
attachment added |
|
my dmesg with ubsan amdgpu traces https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2039368/+attachment/5722326/+files/dmesg-ubsan-amdgpu.txt |
|
| 2023-11-25 12:26:23 |
KonishchevDmitry |
bug |
|
|
added subscriber KonishchevDmitry |
| 2024-02-17 13:05:01 |
john bos |
bug |
|
|
added subscriber john bos |
| 2024-04-04 10:48:25 |
John Anderson |
bug |
|
|
added subscriber John Anderson |
| 2024-06-25 13:41:16 |
Juerg Haefliger |
nominated for series |
|
Ubuntu Noble |
|
| 2024-06-25 13:41:16 |
Juerg Haefliger |
bug task added |
|
linux (Ubuntu Noble) |
|
| 2024-06-25 13:41:25 |
Juerg Haefliger |
linux (Ubuntu Noble): status |
New |
Confirmed |
|
| 2024-06-25 13:41:29 |
Juerg Haefliger |
linux (Ubuntu): status |
Confirmed |
Triaged |
|
| 2024-06-25 13:41:37 |
Juerg Haefliger |
nominated for series |
|
Ubuntu Mantic |
|
| 2024-06-25 13:41:37 |
Juerg Haefliger |
bug task added |
|
linux (Ubuntu Mantic) |
|
| 2024-06-25 13:41:43 |
Juerg Haefliger |
linux (Ubuntu Mantic): status |
New |
Won't Fix |
|
| 2024-06-25 14:58:13 |
Jonathan Crooke |
removed subscriber Jonathan Crooke |
|
|
|
| 2024-07-01 15:08:06 |
Ghadi Rahme |
description |
Since I upgraded from lunar to mantic I get a load of those errors (41 on a fresh boot) in dmesg:
```
[ 4.277343] UBSAN: array-index-out-of-bounds in /build/linux-D15vQj/linux-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 4.277728] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.277925] CPU: 7 PID: 247 Comm: kworker/7:1 Not tainted 6.5.0-9-generic #9-Ubuntu
[ 4.278132] Hardware name: Default string Default string/Default string, BIOS WRX80SU8-F6 06/08/2023
[ 4.278531] Workqueue: events register_cache_worker [bcache]
[ 4.278754] Call Trace:
[ 4.278949] <TASK>
[ 4.279143] dump_stack_lvl+0x48/0x70
[ 4.279337] dump_stack+0x10/0x20
[ 4.279526] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.279721] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 4.279929] bch_btree_node_read_done+0xcb/0x410 [bcache]
[ 4.280142] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.280349] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.280557] bch_btree_node_get.part.0+0x15c/0x330 [bcache]
[ 4.280764] ? __bch_btree_ptr_invalid+0x66/0xe0 [bcache]
[ 4.280975] ? __pfx_up_write+0x10/0x10
[ 4.281170] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.281375] run_cache_set+0x596/0x850 [bcache]
[ 4.281578] ? srso_return_thunk+0x5/0x10
[ 4.281773] register_cache_set+0x1a2/0x210 [bcache]
[ 4.281984] register_cache+0x11a/0x1a0 [bcache]
[ 4.282187] register_cache_worker+0x22/0x80 [bcache]
[ 4.282387] process_one_work+0x223/0x440
[ 4.282573] worker_thread+0x4d/0x3f0
[ 4.282753] ? srso_return_thunk+0x5/0x10
[ 4.282931] ? _raw_spin_lock_irqsave+0xe/0x20
[ 4.283113] ? __pfx_worker_thread+0x10/0x10
[ 4.283286] kthread+0xf2/0x120
[ 4.283458] ? __pfx_kthread+0x10/0x10
[ 4.283631] ret_from_fork+0x47/0x70
[ 4.283800] ? __pfx_kthread+0x10/0x10
[ 4.283972] ret_from_fork_asm+0x1b/0x30
[ 4.284143] </TASK>
```
This system has 4 bcache backing devices and 4 bcache cache devices, though they are not associated for now and caching is disabled. It was already like that when I upgraded, so the kernel only uses the backing code, not the caching one.
ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: linux-image-6.5.0-9-generic 6.5.0-9.9
ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
Uname: Linux 6.5.0-9-generic x86_64
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: GNOME
Date: Sat Oct 14 23:16:33 2023
HibernationDevice: RESUME=none
MachineType: {report['dmi.sys.vendor']} {report['dmi.product.name']}
ProcFB:
0 amdgpudrmfb
1 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/@/boot/vmlinuz-6.5.0-9-generic root=UUID=f35ecf77-511e-4dde-ac11-c1d848e97315 ro rootflags=subvol=@ amdgpu.si_support=1 radeon.si_support=0 amdgpu.cik_support=1 radeon.cik_support=0 amdgpu.exp_hw_support=1 amdgpu.gpu_recovery=1 amdgpu.ppfeaturemask=0xffffffff delayacct zswap.enabled=1
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
linux-restricted-modules-6.5.0-9-generic N/A
linux-backports-modules-6.5.0-9-generic N/A
linux-firmware 20230919.git3672ccab-0ubuntu2.1
RfKill:
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/08/2023
dmi.bios.release: 5.23
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: WRX80SU8-F6
dmi.board.asset.tag: Default string
dmi.board.name: Default string
dmi.board.vendor: Default string
dmi.board.version: Default string
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInternational,LLC.:bvrWRX80SU8-F6:bd06/08/2023:br5.23:svnDefaultstring:pnDefaultstring:pvrDefaultstring:rvnDefaultstring:rnDefaultstring:rvrDefaultstring:cvnDefaultstring:ct3:cvrDefaultstring:skuDefaultstring:
dmi.product.family: Default string
dmi.product.name: Default string
dmi.product.sku: Default string
dmi.product.version: Default string
dmi.sys.vendor: Default string
modified.conffile..etc.default.apport: [modified]
mtime.conffile..etc.default.apport: 2018-06-16T17:39:00.798346 |
[Impact]
Currently there are UBSAN warnings that show up when running bcache on jammy HWE, Mantic and noble. For now no side effects have been observed but such an issue could potentially cause a crash or corrupt data.
[Fix]
There is currently a fix upstream provided by the following patch:
* 3a861560ccb3 "bcache: fix variable length array abuse in btree_iter"
[Test Case]
- Setup bcache on a jammy HWE kernel or mantic or noble machine.
This can be done following the steps in this wiki: https://wiki.ubuntu.com/ServerTeam/Bcache
- Restart the machine
- After restarting the machine UBSAN warnings and call traces can be seen in dmesg.
[Where problems could occur]
-The patch modifies the way bcache allocates space to the btree iterator. The main problems that could occur are different UBSAN warnings showing up that could possibly trigger a crash much easier than the current array index-out-of-bounds being observed.
Thank you @illwieckz for the original bug report
[original description]
Since I upgraded from lunar to mantic I get a load of those errors (41 on a fresh boot) in dmesg:
```
[ 4.277343] UBSAN: array-index-out-of-bounds in /build/linux-D15vQj/linux-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 4.277728] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.277925] CPU: 7 PID: 247 Comm: kworker/7:1 Not tainted 6.5.0-9-generic #9-Ubuntu
[ 4.278132] Hardware name: Default string Default string/Default string, BIOS WRX80SU8-F6 06/08/2023
[ 4.278531] Workqueue: events register_cache_worker [bcache]
[ 4.278754] Call Trace:
[ 4.278949] <TASK>
[ 4.279143] dump_stack_lvl+0x48/0x70
[ 4.279337] dump_stack+0x10/0x20
[ 4.279526] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.279721] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 4.279929] bch_btree_node_read_done+0xcb/0x410 [bcache]
[ 4.280142] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.280349] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.280557] bch_btree_node_get.part.0+0x15c/0x330 [bcache]
[ 4.280764] ? __bch_btree_ptr_invalid+0x66/0xe0 [bcache]
[ 4.280975] ? __pfx_up_write+0x10/0x10
[ 4.281170] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.281375] run_cache_set+0x596/0x850 [bcache]
[ 4.281578] ? srso_return_thunk+0x5/0x10
[ 4.281773] register_cache_set+0x1a2/0x210 [bcache]
[ 4.281984] register_cache+0x11a/0x1a0 [bcache]
[ 4.282187] register_cache_worker+0x22/0x80 [bcache]
[ 4.282387] process_one_work+0x223/0x440
[ 4.282573] worker_thread+0x4d/0x3f0
[ 4.282753] ? srso_return_thunk+0x5/0x10
[ 4.282931] ? _raw_spin_lock_irqsave+0xe/0x20
[ 4.283113] ? __pfx_worker_thread+0x10/0x10
[ 4.283286] kthread+0xf2/0x120
[ 4.283458] ? __pfx_kthread+0x10/0x10
[ 4.283631] ret_from_fork+0x47/0x70
[ 4.283800] ? __pfx_kthread+0x10/0x10
[ 4.283972] ret_from_fork_asm+0x1b/0x30
[ 4.284143] </TASK>
```
This system has 4 bcache backing devices and 4 bcache cache devices, though they are not associated for now and caching is disabled. It was already like that when I upgraded, so the kernel only uses the backing code, not the caching one.
ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: linux-image-6.5.0-9-generic 6.5.0-9.9
ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
Uname: Linux 6.5.0-9-generic x86_64
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: GNOME
Date: Sat Oct 14 23:16:33 2023
HibernationDevice: RESUME=none
MachineType: {report['dmi.sys.vendor']} {report['dmi.product.name']}
ProcFB:
0 amdgpudrmfb
1 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/@/boot/vmlinuz-6.5.0-9-generic root=UUID=f35ecf77-511e-4dde-ac11-c1d848e97315 ro rootflags=subvol=@ amdgpu.si_support=1 radeon.si_support=0 amdgpu.cik_support=1 radeon.cik_support=0 amdgpu.exp_hw_support=1 amdgpu.gpu_recovery=1 amdgpu.ppfeaturemask=0xffffffff delayacct zswap.enabled=1
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
linux-restricted-modules-6.5.0-9-generic N/A
linux-backports-modules-6.5.0-9-generic N/A
linux-firmware 20230919.git3672ccab-0ubuntu2.1
RfKill:
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/08/2023
dmi.bios.release: 5.23
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: WRX80SU8-F6
dmi.board.asset.tag: Default string
dmi.board.name: Default string
dmi.board.vendor: Default string
dmi.board.version: Default string
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInternational,LLC.:bvrWRX80SU8-F6:bd06/08/2023:br5.23:svnDefaultstring:pnDefaultstring:pvrDefaultstring:rvnDefaultstring:rnDefaultstring:rvrDefaultstring:cvnDefaultstring:ct3:cvrDefaultstring:skuDefaultstring:
dmi.product.family: Default string
dmi.product.name: Default string
dmi.product.sku: Default string
dmi.product.version: Default string
dmi.sys.vendor: Default string
modified.conffile..etc.default.apport: [modified]
mtime.conffile..etc.default.apport: 2018-06-16T17:39:00.798346 |
|
| 2024-07-01 15:17:42 |
Ghadi Rahme |
nominated for series |
|
Ubuntu Jammy |
|
| 2024-07-01 15:17:42 |
Ghadi Rahme |
bug task added |
|
linux (Ubuntu Jammy) |
|
| 2024-07-01 15:17:42 |
Ghadi Rahme |
nominated for series |
|
Ubuntu Oracular |
|
| 2024-07-01 15:17:42 |
Ghadi Rahme |
bug task added |
|
linux (Ubuntu Oracular) |
|
| 2024-07-01 20:33:30 |
Ghadi Rahme |
description |
[Impact]
Currently there are UBSAN warnings that show up when running bcache on jammy HWE, Mantic and noble. For now no side effects have been observed but such an issue could potentially cause a crash or corrupt data.
[Fix]
There is currently a fix upstream provided by the following patch:
* 3a861560ccb3 "bcache: fix variable length array abuse in btree_iter"
[Test Case]
- Setup bcache on a jammy HWE kernel or mantic or noble machine.
This can be done following the steps in this wiki: https://wiki.ubuntu.com/ServerTeam/Bcache
- Restart the machine
- After restarting the machine UBSAN warnings and call traces can be seen in dmesg.
[Where problems could occur]
-The patch modifies the way bcache allocates space to the btree iterator. The main problems that could occur are different UBSAN warnings showing up that could possibly trigger a crash much easier than the current array index-out-of-bounds being observed.
Thank you @illwieckz for the original bug report
[original description]
Since I upgraded from lunar to mantic I get a load of those errors (41 on a fresh boot) in dmesg:
```
[ 4.277343] UBSAN: array-index-out-of-bounds in /build/linux-D15vQj/linux-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 4.277728] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.277925] CPU: 7 PID: 247 Comm: kworker/7:1 Not tainted 6.5.0-9-generic #9-Ubuntu
[ 4.278132] Hardware name: Default string Default string/Default string, BIOS WRX80SU8-F6 06/08/2023
[ 4.278531] Workqueue: events register_cache_worker [bcache]
[ 4.278754] Call Trace:
[ 4.278949] <TASK>
[ 4.279143] dump_stack_lvl+0x48/0x70
[ 4.279337] dump_stack+0x10/0x20
[ 4.279526] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.279721] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 4.279929] bch_btree_node_read_done+0xcb/0x410 [bcache]
[ 4.280142] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.280349] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.280557] bch_btree_node_get.part.0+0x15c/0x330 [bcache]
[ 4.280764] ? __bch_btree_ptr_invalid+0x66/0xe0 [bcache]
[ 4.280975] ? __pfx_up_write+0x10/0x10
[ 4.281170] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.281375] run_cache_set+0x596/0x850 [bcache]
[ 4.281578] ? srso_return_thunk+0x5/0x10
[ 4.281773] register_cache_set+0x1a2/0x210 [bcache]
[ 4.281984] register_cache+0x11a/0x1a0 [bcache]
[ 4.282187] register_cache_worker+0x22/0x80 [bcache]
[ 4.282387] process_one_work+0x223/0x440
[ 4.282573] worker_thread+0x4d/0x3f0
[ 4.282753] ? srso_return_thunk+0x5/0x10
[ 4.282931] ? _raw_spin_lock_irqsave+0xe/0x20
[ 4.283113] ? __pfx_worker_thread+0x10/0x10
[ 4.283286] kthread+0xf2/0x120
[ 4.283458] ? __pfx_kthread+0x10/0x10
[ 4.283631] ret_from_fork+0x47/0x70
[ 4.283800] ? __pfx_kthread+0x10/0x10
[ 4.283972] ret_from_fork_asm+0x1b/0x30
[ 4.284143] </TASK>
```
This system has 4 bcache backing devices and 4 bcache cache devices, though they are not associated for now and caching is disabled. It was already like that when I upgraded, so the kernel only uses the backing code, not the caching one.
ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: linux-image-6.5.0-9-generic 6.5.0-9.9
ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
Uname: Linux 6.5.0-9-generic x86_64
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: GNOME
Date: Sat Oct 14 23:16:33 2023
HibernationDevice: RESUME=none
MachineType: {report['dmi.sys.vendor']} {report['dmi.product.name']}
ProcFB:
0 amdgpudrmfb
1 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/@/boot/vmlinuz-6.5.0-9-generic root=UUID=f35ecf77-511e-4dde-ac11-c1d848e97315 ro rootflags=subvol=@ amdgpu.si_support=1 radeon.si_support=0 amdgpu.cik_support=1 radeon.cik_support=0 amdgpu.exp_hw_support=1 amdgpu.gpu_recovery=1 amdgpu.ppfeaturemask=0xffffffff delayacct zswap.enabled=1
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
linux-restricted-modules-6.5.0-9-generic N/A
linux-backports-modules-6.5.0-9-generic N/A
linux-firmware 20230919.git3672ccab-0ubuntu2.1
RfKill:
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/08/2023
dmi.bios.release: 5.23
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: WRX80SU8-F6
dmi.board.asset.tag: Default string
dmi.board.name: Default string
dmi.board.vendor: Default string
dmi.board.version: Default string
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInternational,LLC.:bvrWRX80SU8-F6:bd06/08/2023:br5.23:svnDefaultstring:pnDefaultstring:pvrDefaultstring:rvnDefaultstring:rnDefaultstring:rvrDefaultstring:cvnDefaultstring:ct3:cvrDefaultstring:skuDefaultstring:
dmi.product.family: Default string
dmi.product.name: Default string
dmi.product.sku: Default string
dmi.product.version: Default string
dmi.sys.vendor: Default string
modified.conffile..etc.default.apport: [modified]
mtime.conffile..etc.default.apport: 2018-06-16T17:39:00.798346 |
[Impact]
Currently there are UBSAN warnings that show up when running bcache on jammy HWE, Mantic and noble. For now no side effects have been observed but such an issue could potentially cause a crash or corrupt data.
[Fix]
There is currently a fix upstream provided by the following patch:
* 3a861560ccb3 "bcache: fix variable length array abuse in btree_iter"
[Test Case]
- Setup bcache on a jammy HWE kernel or mantic or noble machine.
This can be done following the steps in this wiki: https://wiki.ubuntu.com/ServerTeam/Bcache
- Restart the machine
- After restarting the machine, the following UBSAN warnings and call traces can be seen in dmesg:
[ 3.824281] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 3.826338] index 4 is out of range for type 'btree_iter_set [4]'
[ 3.826812] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 3.827817] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 3.828835] Workqueue: events register_cache_worker [bcache]
[ 3.829429] Call Trace:
[ 3.830626] <TASK>
[ 3.831638] dump_stack_lvl+0x48/0x70
[ 3.832227] dump_stack+0x10/0x20
[ 3.832785] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 3.833357] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 3.834052] bch_btree_node_read_done+0xfc/0x450 [bcache]
[ 3.834653] ? mempool_kfree+0xe/0x20
[ 3.835211] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 3.835832] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 3.836474] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 3.837161] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 3.837838] ? __pfx_up_write+0x10/0x10
[ 3.838739] bch_btree_node_get+0x16/0x30 [bcache]
[ 3.839506] run_cache_set+0x596/0x840 [bcache]
[ 3.840197] register_cache_set+0x1a2/0x210 [bcache]
[ 3.840748] register_cache+0x11a/0x1a0 [bcache]
[ 3.841303] register_cache_worker+0x22/0x80 [bcache]
[ 3.841840] process_one_work+0x23d/0x450
[ 3.842297] worker_thread+0x50/0x3f0
[ 3.842698] ? __pfx_worker_thread+0x10/0x10
[ 3.843081] kthread+0xef/0x120
[ 3.843521] ? __pfx_kthread+0x10/0x10
[ 3.843892] ret_from_fork+0x44/0x70
[ 3.844264] ? __pfx_kthread+0x10/0x10
[ 3.844611] ret_from_fork_asm+0x1b/0x30
[ 3.844949] </TASK>
[ 4.029242] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1207:3
[ 4.030496] index 14 is out of range for type 'btree_iter_set [4]'
[ 4.030930] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.031841] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.032650] Workqueue: events register_cache_worker [bcache]
[ 4.033149] Call Trace:
[ 4.033549] <TASK>
[ 4.033972] dump_stack_lvl+0x48/0x70
[ 4.034418] dump_stack+0x10/0x20
[ 4.034839] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.035279] btree_mergesort+0x4d4/0x520 [bcache]
[ 4.035730] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.036191] ? __pfx_bch_extent_sort_cmp+0x10/0x10 [bcache]
[ 4.036691] __btree_sort+0x96/0x2d0 [bcache]
[ 4.037182] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.037674] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.038172] ? mempool_kfree+0xe/0x20
[ 4.038617] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.039120] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.039659] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.040220] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.040806] ? __pfx_up_write+0x10/0x10
[ 4.041371] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.041921] run_cache_set+0x596/0x840 [bcache]
[ 4.042497] register_cache_set+0x1a2/0x210 [bcache]
[ 4.043089] register_cache+0x11a/0x1a0 [bcache]
[ 4.043715] register_cache_worker+0x22/0x80 [bcache]
[ 4.044348] process_one_work+0x23d/0x450
[ 4.044887] worker_thread+0x50/0x3f0
[ 4.045422] ? __pfx_worker_thread+0x10/0x10
[ 4.045936] kthread+0xef/0x120
[ 4.046445] ? __pfx_kthread+0x10/0x10
[ 4.046942] ret_from_fork+0x44/0x70
[ 4.047423] ? __pfx_kthread+0x10/0x10
[ 4.047878] ret_from_fork_asm+0x1b/0x30
[ 4.048339] </TASK>
[ 4.227653] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:281:4
[ 4.228847] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.229472] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.230680] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.231954] Workqueue: events register_cache_worker [bcache]
[ 4.232690] Call Trace:
[ 4.233327] <TASK>
[ 4.233935] dump_stack_lvl+0x48/0x70
[ 4.234568] dump_stack+0x10/0x20
[ 4.235219] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.235833] bch_extent_sort_fixup+0xb95/0xd70 [bcache]
[ 4.236524] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.237159] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.237839] btree_mergesort+0x221/0x520 [bcache]
[ 4.238823] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.239800] __btree_sort+0x96/0x2d0 [bcache]
[ 4.240880] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.243046] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.245223] ? mempool_kfree+0xe/0x20
[ 4.246311] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.247410] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.248471] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.248959] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.249454] ? __pfx_up_write+0x10/0x10
[ 4.249904] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.250386] run_cache_set+0x596/0x840 [bcache]
[ 4.250842] register_cache_set+0x1a2/0x210 [bcache]
[ 4.251319] register_cache+0x11a/0x1a0 [bcache]
[ 4.251748] register_cache_worker+0x22/0x80 [bcache]
[ 4.252181] process_one_work+0x23d/0x450
[ 4.252559] worker_thread+0x50/0x3f0
[ 4.252922] ? __pfx_worker_thread+0x10/0x10
[ 4.253286] kthread+0xef/0x120
[ 4.253659] ? __pfx_kthread+0x10/0x10
[ 4.254024] ret_from_fork+0x44/0x70
[ 4.254394] ? __pfx_kthread+0x10/0x10
[ 4.254755] ret_from_fork_asm+0x1b/0x30
[ 4.255145] </TASK>
[ 4.257388] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:36:18
[ 4.258429] index 14 is out of range for type 'btree_iter_set [4]'
[ 4.258964] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.260073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.261188] Workqueue: events register_cache_worker [bcache]
[ 4.261811] Call Trace:
[ 4.262374] <TASK>
[ 4.262912] dump_stack_lvl+0x48/0x70
[ 4.263502] dump_stack+0x10/0x20
[ 4.264042] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.264605] bch_extent_sort_fixup+0xbe5/0xd70 [bcache]
[ 4.265218] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.265821] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.266514] btree_mergesort+0x221/0x520 [bcache]
[ 4.267234] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.267882] __btree_sort+0x96/0x2d0 [bcache]
[ 4.268508] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.269144] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.269825] ? mempool_kfree+0xe/0x20
[ 4.270489] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.271243] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.272293] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.273260] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.274182] ? __pfx_up_write+0x10/0x10
[ 4.274973] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.276053] run_cache_set+0x596/0x840 [bcache]
[ 4.276972] register_cache_set+0x1a2/0x210 [bcache]
[ 4.277865] register_cache+0x11a/0x1a0 [bcache]
[ 4.278703] register_cache_worker+0x22/0x80 [bcache]
[ 4.279907] process_one_work+0x23d/0x450
[ 4.280690] worker_thread+0x50/0x3f0
[ 4.282228] ? __pfx_worker_thread+0x10/0x10
[ 4.283082] kthread+0xef/0x120
[ 4.283467] ? __pfx_kthread+0x10/0x10
[ 4.283803] ret_from_fork+0x44/0x70
[ 4.284143] ? __pfx_kthread+0x10/0x10
[ 4.284474] ret_from_fork_asm+0x1b/0x30
[ 4.284807] </TASK>
[ 4.286129] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:291:4
[ 4.286791] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.287231] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.288033] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.288863] Workqueue: events register_cache_worker [bcache]
[ 4.289340] Call Trace:
[ 4.289753] <TASK>
[ 4.290168] dump_stack_lvl+0x48/0x70
[ 4.290581] dump_stack+0x10/0x20
[ 4.290984] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.291432] bch_extent_sort_fixup+0xb77/0xd70 [bcache]
[ 4.291882] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.292309] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.292764] btree_mergesort+0x221/0x520 [bcache]
[ 4.293225] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.293683] __btree_sort+0x96/0x2d0 [bcache]
[ 4.294153] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.294631] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.295175] ? mempool_kfree+0xe/0x20
[ 4.295671] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.296257] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.296834] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.297446] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.298087] ? __pfx_up_write+0x10/0x10
[ 4.298678] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.299336] run_cache_set+0x596/0x840 [bcache]
[ 4.299941] register_cache_set+0x1a2/0x210 [bcache]
[ 4.300556] register_cache+0x11a/0x1a0 [bcache]
[ 4.301257] register_cache_worker+0x22/0x80 [bcache]
[ 4.302031] process_one_work+0x23d/0x450
[ 4.302722] worker_thread+0x50/0x3f0
[ 4.303410] ? __pfx_worker_thread+0x10/0x10
[ 4.304008] kthread+0xef/0x120
[ 4.304529] ? __pfx_kthread+0x10/0x10
[ 4.304910] ret_from_fork+0x44/0x70
[ 4.305315] ? __pfx_kthread+0x10/0x10
[ 4.305690] ret_from_fork_asm+0x1b/0x30
[ 4.306037] </TASK>
[Where problems could occur]
-The patch modifies the way bcache allocates space to the btree iterator. The main problems that could occur are different UBSAN warnings showing up that could possibly trigger a crash much easier than the current array index-out-of-bounds being observed.
Thank you @illwieckz for the original bug report
[original description]
Since I upgraded from lunar to mantic I get a load of those errors (41 on a fresh boot) in dmesg:
```
[ 4.277343] UBSAN: array-index-out-of-bounds in /build/linux-D15vQj/linux-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 4.277728] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.277925] CPU: 7 PID: 247 Comm: kworker/7:1 Not tainted 6.5.0-9-generic #9-Ubuntu
[ 4.278132] Hardware name: Default string Default string/Default string, BIOS WRX80SU8-F6 06/08/2023
[ 4.278531] Workqueue: events register_cache_worker [bcache]
[ 4.278754] Call Trace:
[ 4.278949] <TASK>
[ 4.279143] dump_stack_lvl+0x48/0x70
[ 4.279337] dump_stack+0x10/0x20
[ 4.279526] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.279721] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 4.279929] bch_btree_node_read_done+0xcb/0x410 [bcache]
[ 4.280142] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.280349] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.280557] bch_btree_node_get.part.0+0x15c/0x330 [bcache]
[ 4.280764] ? __bch_btree_ptr_invalid+0x66/0xe0 [bcache]
[ 4.280975] ? __pfx_up_write+0x10/0x10
[ 4.281170] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.281375] run_cache_set+0x596/0x850 [bcache]
[ 4.281578] ? srso_return_thunk+0x5/0x10
[ 4.281773] register_cache_set+0x1a2/0x210 [bcache]
[ 4.281984] register_cache+0x11a/0x1a0 [bcache]
[ 4.282187] register_cache_worker+0x22/0x80 [bcache]
[ 4.282387] process_one_work+0x223/0x440
[ 4.282573] worker_thread+0x4d/0x3f0
[ 4.282753] ? srso_return_thunk+0x5/0x10
[ 4.282931] ? _raw_spin_lock_irqsave+0xe/0x20
[ 4.283113] ? __pfx_worker_thread+0x10/0x10
[ 4.283286] kthread+0xf2/0x120
[ 4.283458] ? __pfx_kthread+0x10/0x10
[ 4.283631] ret_from_fork+0x47/0x70
[ 4.283800] ? __pfx_kthread+0x10/0x10
[ 4.283972] ret_from_fork_asm+0x1b/0x30
[ 4.284143] </TASK>
```
This system has 4 bcache backing devices and 4 bcache cache devices, though they are not associated for now and caching is disabled. It was already like that when I upgraded, so the kernel only uses the backing code, not the caching one.
ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: linux-image-6.5.0-9-generic 6.5.0-9.9
ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
Uname: Linux 6.5.0-9-generic x86_64
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: GNOME
Date: Sat Oct 14 23:16:33 2023
HibernationDevice: RESUME=none
MachineType: {report['dmi.sys.vendor']} {report['dmi.product.name']}
ProcFB:
0 amdgpudrmfb
1 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/@/boot/vmlinuz-6.5.0-9-generic root=UUID=f35ecf77-511e-4dde-ac11-c1d848e97315 ro rootflags=subvol=@ amdgpu.si_support=1 radeon.si_support=0 amdgpu.cik_support=1 radeon.cik_support=0 amdgpu.exp_hw_support=1 amdgpu.gpu_recovery=1 amdgpu.ppfeaturemask=0xffffffff delayacct zswap.enabled=1
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
linux-restricted-modules-6.5.0-9-generic N/A
linux-backports-modules-6.5.0-9-generic N/A
linux-firmware 20230919.git3672ccab-0ubuntu2.1
RfKill:
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/08/2023
dmi.bios.release: 5.23
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: WRX80SU8-F6
dmi.board.asset.tag: Default string
dmi.board.name: Default string
dmi.board.vendor: Default string
dmi.board.version: Default string
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInternational,LLC.:bvrWRX80SU8-F6:bd06/08/2023:br5.23:svnDefaultstring:pnDefaultstring:pvrDefaultstring:rvnDefaultstring:rnDefaultstring:rvrDefaultstring:cvnDefaultstring:ct3:cvrDefaultstring:skuDefaultstring:
dmi.product.family: Default string
dmi.product.name: Default string
dmi.product.sku: Default string
dmi.product.version: Default string
dmi.sys.vendor: Default string
modified.conffile..etc.default.apport: [modified]
mtime.conffile..etc.default.apport: 2018-06-16T17:39:00.798346 |
|
| 2024-07-01 20:33:53 |
Ghadi Rahme |
description |
[Impact]
Currently there are UBSAN warnings that show up when running bcache on jammy HWE, Mantic and noble. For now no side effects have been observed but such an issue could potentially cause a crash or corrupt data.
[Fix]
There is currently a fix upstream provided by the following patch:
* 3a861560ccb3 "bcache: fix variable length array abuse in btree_iter"
[Test Case]
- Setup bcache on a jammy HWE kernel or mantic or noble machine.
This can be done following the steps in this wiki: https://wiki.ubuntu.com/ServerTeam/Bcache
- Restart the machine
- After restarting the machine, the following UBSAN warnings and call traces can be seen in dmesg:
[ 3.824281] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 3.826338] index 4 is out of range for type 'btree_iter_set [4]'
[ 3.826812] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 3.827817] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 3.828835] Workqueue: events register_cache_worker [bcache]
[ 3.829429] Call Trace:
[ 3.830626] <TASK>
[ 3.831638] dump_stack_lvl+0x48/0x70
[ 3.832227] dump_stack+0x10/0x20
[ 3.832785] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 3.833357] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 3.834052] bch_btree_node_read_done+0xfc/0x450 [bcache]
[ 3.834653] ? mempool_kfree+0xe/0x20
[ 3.835211] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 3.835832] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 3.836474] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 3.837161] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 3.837838] ? __pfx_up_write+0x10/0x10
[ 3.838739] bch_btree_node_get+0x16/0x30 [bcache]
[ 3.839506] run_cache_set+0x596/0x840 [bcache]
[ 3.840197] register_cache_set+0x1a2/0x210 [bcache]
[ 3.840748] register_cache+0x11a/0x1a0 [bcache]
[ 3.841303] register_cache_worker+0x22/0x80 [bcache]
[ 3.841840] process_one_work+0x23d/0x450
[ 3.842297] worker_thread+0x50/0x3f0
[ 3.842698] ? __pfx_worker_thread+0x10/0x10
[ 3.843081] kthread+0xef/0x120
[ 3.843521] ? __pfx_kthread+0x10/0x10
[ 3.843892] ret_from_fork+0x44/0x70
[ 3.844264] ? __pfx_kthread+0x10/0x10
[ 3.844611] ret_from_fork_asm+0x1b/0x30
[ 3.844949] </TASK>
[ 4.029242] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1207:3
[ 4.030496] index 14 is out of range for type 'btree_iter_set [4]'
[ 4.030930] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.031841] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.032650] Workqueue: events register_cache_worker [bcache]
[ 4.033149] Call Trace:
[ 4.033549] <TASK>
[ 4.033972] dump_stack_lvl+0x48/0x70
[ 4.034418] dump_stack+0x10/0x20
[ 4.034839] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.035279] btree_mergesort+0x4d4/0x520 [bcache]
[ 4.035730] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.036191] ? __pfx_bch_extent_sort_cmp+0x10/0x10 [bcache]
[ 4.036691] __btree_sort+0x96/0x2d0 [bcache]
[ 4.037182] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.037674] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.038172] ? mempool_kfree+0xe/0x20
[ 4.038617] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.039120] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.039659] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.040220] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.040806] ? __pfx_up_write+0x10/0x10
[ 4.041371] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.041921] run_cache_set+0x596/0x840 [bcache]
[ 4.042497] register_cache_set+0x1a2/0x210 [bcache]
[ 4.043089] register_cache+0x11a/0x1a0 [bcache]
[ 4.043715] register_cache_worker+0x22/0x80 [bcache]
[ 4.044348] process_one_work+0x23d/0x450
[ 4.044887] worker_thread+0x50/0x3f0
[ 4.045422] ? __pfx_worker_thread+0x10/0x10
[ 4.045936] kthread+0xef/0x120
[ 4.046445] ? __pfx_kthread+0x10/0x10
[ 4.046942] ret_from_fork+0x44/0x70
[ 4.047423] ? __pfx_kthread+0x10/0x10
[ 4.047878] ret_from_fork_asm+0x1b/0x30
[ 4.048339] </TASK>
[ 4.227653] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:281:4
[ 4.228847] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.229472] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.230680] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.231954] Workqueue: events register_cache_worker [bcache]
[ 4.232690] Call Trace:
[ 4.233327] <TASK>
[ 4.233935] dump_stack_lvl+0x48/0x70
[ 4.234568] dump_stack+0x10/0x20
[ 4.235219] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.235833] bch_extent_sort_fixup+0xb95/0xd70 [bcache]
[ 4.236524] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.237159] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.237839] btree_mergesort+0x221/0x520 [bcache]
[ 4.238823] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.239800] __btree_sort+0x96/0x2d0 [bcache]
[ 4.240880] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.243046] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.245223] ? mempool_kfree+0xe/0x20
[ 4.246311] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.247410] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.248471] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.248959] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.249454] ? __pfx_up_write+0x10/0x10
[ 4.249904] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.250386] run_cache_set+0x596/0x840 [bcache]
[ 4.250842] register_cache_set+0x1a2/0x210 [bcache]
[ 4.251319] register_cache+0x11a/0x1a0 [bcache]
[ 4.251748] register_cache_worker+0x22/0x80 [bcache]
[ 4.252181] process_one_work+0x23d/0x450
[ 4.252559] worker_thread+0x50/0x3f0
[ 4.252922] ? __pfx_worker_thread+0x10/0x10
[ 4.253286] kthread+0xef/0x120
[ 4.253659] ? __pfx_kthread+0x10/0x10
[ 4.254024] ret_from_fork+0x44/0x70
[ 4.254394] ? __pfx_kthread+0x10/0x10
[ 4.254755] ret_from_fork_asm+0x1b/0x30
[ 4.255145] </TASK>
[ 4.257388] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:36:18
[ 4.258429] index 14 is out of range for type 'btree_iter_set [4]'
[ 4.258964] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.260073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.261188] Workqueue: events register_cache_worker [bcache]
[ 4.261811] Call Trace:
[ 4.262374] <TASK>
[ 4.262912] dump_stack_lvl+0x48/0x70
[ 4.263502] dump_stack+0x10/0x20
[ 4.264042] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.264605] bch_extent_sort_fixup+0xbe5/0xd70 [bcache]
[ 4.265218] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.265821] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.266514] btree_mergesort+0x221/0x520 [bcache]
[ 4.267234] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.267882] __btree_sort+0x96/0x2d0 [bcache]
[ 4.268508] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.269144] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.269825] ? mempool_kfree+0xe/0x20
[ 4.270489] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.271243] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.272293] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.273260] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.274182] ? __pfx_up_write+0x10/0x10
[ 4.274973] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.276053] run_cache_set+0x596/0x840 [bcache]
[ 4.276972] register_cache_set+0x1a2/0x210 [bcache]
[ 4.277865] register_cache+0x11a/0x1a0 [bcache]
[ 4.278703] register_cache_worker+0x22/0x80 [bcache]
[ 4.279907] process_one_work+0x23d/0x450
[ 4.280690] worker_thread+0x50/0x3f0
[ 4.282228] ? __pfx_worker_thread+0x10/0x10
[ 4.283082] kthread+0xef/0x120
[ 4.283467] ? __pfx_kthread+0x10/0x10
[ 4.283803] ret_from_fork+0x44/0x70
[ 4.284143] ? __pfx_kthread+0x10/0x10
[ 4.284474] ret_from_fork_asm+0x1b/0x30
[ 4.284807] </TASK>
[ 4.286129] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:291:4
[ 4.286791] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.287231] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.288033] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.288863] Workqueue: events register_cache_worker [bcache]
[ 4.289340] Call Trace:
[ 4.289753] <TASK>
[ 4.290168] dump_stack_lvl+0x48/0x70
[ 4.290581] dump_stack+0x10/0x20
[ 4.290984] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.291432] bch_extent_sort_fixup+0xb77/0xd70 [bcache]
[ 4.291882] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.292309] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.292764] btree_mergesort+0x221/0x520 [bcache]
[ 4.293225] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.293683] __btree_sort+0x96/0x2d0 [bcache]
[ 4.294153] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.294631] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.295175] ? mempool_kfree+0xe/0x20
[ 4.295671] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.296257] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.296834] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.297446] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.298087] ? __pfx_up_write+0x10/0x10
[ 4.298678] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.299336] run_cache_set+0x596/0x840 [bcache]
[ 4.299941] register_cache_set+0x1a2/0x210 [bcache]
[ 4.300556] register_cache+0x11a/0x1a0 [bcache]
[ 4.301257] register_cache_worker+0x22/0x80 [bcache]
[ 4.302031] process_one_work+0x23d/0x450
[ 4.302722] worker_thread+0x50/0x3f0
[ 4.303410] ? __pfx_worker_thread+0x10/0x10
[ 4.304008] kthread+0xef/0x120
[ 4.304529] ? __pfx_kthread+0x10/0x10
[ 4.304910] ret_from_fork+0x44/0x70
[ 4.305315] ? __pfx_kthread+0x10/0x10
[ 4.305690] ret_from_fork_asm+0x1b/0x30
[ 4.306037] </TASK>
[Where problems could occur]
-The patch modifies the way bcache allocates space to the btree iterator. The main problems that could occur are different UBSAN warnings showing up that could possibly trigger a crash much easier than the current array index-out-of-bounds being observed.
Thank you @illwieckz for the original bug report
[original description]
Since I upgraded from lunar to mantic I get a load of those errors (41 on a fresh boot) in dmesg:
```
[ 4.277343] UBSAN: array-index-out-of-bounds in /build/linux-D15vQj/linux-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 4.277728] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.277925] CPU: 7 PID: 247 Comm: kworker/7:1 Not tainted 6.5.0-9-generic #9-Ubuntu
[ 4.278132] Hardware name: Default string Default string/Default string, BIOS WRX80SU8-F6 06/08/2023
[ 4.278531] Workqueue: events register_cache_worker [bcache]
[ 4.278754] Call Trace:
[ 4.278949] <TASK>
[ 4.279143] dump_stack_lvl+0x48/0x70
[ 4.279337] dump_stack+0x10/0x20
[ 4.279526] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.279721] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 4.279929] bch_btree_node_read_done+0xcb/0x410 [bcache]
[ 4.280142] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.280349] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.280557] bch_btree_node_get.part.0+0x15c/0x330 [bcache]
[ 4.280764] ? __bch_btree_ptr_invalid+0x66/0xe0 [bcache]
[ 4.280975] ? __pfx_up_write+0x10/0x10
[ 4.281170] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.281375] run_cache_set+0x596/0x850 [bcache]
[ 4.281578] ? srso_return_thunk+0x5/0x10
[ 4.281773] register_cache_set+0x1a2/0x210 [bcache]
[ 4.281984] register_cache+0x11a/0x1a0 [bcache]
[ 4.282187] register_cache_worker+0x22/0x80 [bcache]
[ 4.282387] process_one_work+0x223/0x440
[ 4.282573] worker_thread+0x4d/0x3f0
[ 4.282753] ? srso_return_thunk+0x5/0x10
[ 4.282931] ? _raw_spin_lock_irqsave+0xe/0x20
[ 4.283113] ? __pfx_worker_thread+0x10/0x10
[ 4.283286] kthread+0xf2/0x120
[ 4.283458] ? __pfx_kthread+0x10/0x10
[ 4.283631] ret_from_fork+0x47/0x70
[ 4.283800] ? __pfx_kthread+0x10/0x10
[ 4.283972] ret_from_fork_asm+0x1b/0x30
[ 4.284143] </TASK>
```
This system has 4 bcache backing devices and 4 bcache cache devices, though they are not associated for now and caching is disabled. It was already like that when I upgraded, so the kernel only uses the backing code, not the caching one.
ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: linux-image-6.5.0-9-generic 6.5.0-9.9
ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
Uname: Linux 6.5.0-9-generic x86_64
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: GNOME
Date: Sat Oct 14 23:16:33 2023
HibernationDevice: RESUME=none
MachineType: {report['dmi.sys.vendor']} {report['dmi.product.name']}
ProcFB:
0 amdgpudrmfb
1 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/@/boot/vmlinuz-6.5.0-9-generic root=UUID=f35ecf77-511e-4dde-ac11-c1d848e97315 ro rootflags=subvol=@ amdgpu.si_support=1 radeon.si_support=0 amdgpu.cik_support=1 radeon.cik_support=0 amdgpu.exp_hw_support=1 amdgpu.gpu_recovery=1 amdgpu.ppfeaturemask=0xffffffff delayacct zswap.enabled=1
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
linux-restricted-modules-6.5.0-9-generic N/A
linux-backports-modules-6.5.0-9-generic N/A
linux-firmware 20230919.git3672ccab-0ubuntu2.1
RfKill:
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/08/2023
dmi.bios.release: 5.23
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: WRX80SU8-F6
dmi.board.asset.tag: Default string
dmi.board.name: Default string
dmi.board.vendor: Default string
dmi.board.version: Default string
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInternational,LLC.:bvrWRX80SU8-F6:bd06/08/2023:br5.23:svnDefaultstring:pnDefaultstring:pvrDefaultstring:rvnDefaultstring:rnDefaultstring:rvrDefaultstring:cvnDefaultstring:ct3:cvrDefaultstring:skuDefaultstring:
dmi.product.family: Default string
dmi.product.name: Default string
dmi.product.sku: Default string
dmi.product.version: Default string
dmi.sys.vendor: Default string
modified.conffile..etc.default.apport: [modified]
mtime.conffile..etc.default.apport: 2018-06-16T17:39:00.798346 |
[Impact]
Currently there are UBSAN warnings that show up when running bcache on jammy HWE, Mantic and noble. For now no side effects have been observed but such an issue could potentially cause a crash or corrupt data.
[Fix]
There is currently a fix upstream provided by the following patch:
* 3a861560ccb3 "bcache: fix variable length array abuse in btree_iter"
[Test Case]
1. Setup bcache on a jammy HWE kernel or mantic or noble machine. This can be done following the steps in this wiki: https://wiki.ubuntu.com/ServerTeam/Bcache
2. Restart the machine
3. After restarting the machine, the following UBSAN warnings and call traces can be seen in dmesg:
[ 3.824281] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 3.826338] index 4 is out of range for type 'btree_iter_set [4]'
[ 3.826812] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 3.827817] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 3.828835] Workqueue: events register_cache_worker [bcache]
[ 3.829429] Call Trace:
[ 3.830626] <TASK>
[ 3.831638] dump_stack_lvl+0x48/0x70
[ 3.832227] dump_stack+0x10/0x20
[ 3.832785] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 3.833357] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 3.834052] bch_btree_node_read_done+0xfc/0x450 [bcache]
[ 3.834653] ? mempool_kfree+0xe/0x20
[ 3.835211] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 3.835832] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 3.836474] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 3.837161] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 3.837838] ? __pfx_up_write+0x10/0x10
[ 3.838739] bch_btree_node_get+0x16/0x30 [bcache]
[ 3.839506] run_cache_set+0x596/0x840 [bcache]
[ 3.840197] register_cache_set+0x1a2/0x210 [bcache]
[ 3.840748] register_cache+0x11a/0x1a0 [bcache]
[ 3.841303] register_cache_worker+0x22/0x80 [bcache]
[ 3.841840] process_one_work+0x23d/0x450
[ 3.842297] worker_thread+0x50/0x3f0
[ 3.842698] ? __pfx_worker_thread+0x10/0x10
[ 3.843081] kthread+0xef/0x120
[ 3.843521] ? __pfx_kthread+0x10/0x10
[ 3.843892] ret_from_fork+0x44/0x70
[ 3.844264] ? __pfx_kthread+0x10/0x10
[ 3.844611] ret_from_fork_asm+0x1b/0x30
[ 3.844949] </TASK>
[ 4.029242] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1207:3
[ 4.030496] index 14 is out of range for type 'btree_iter_set [4]'
[ 4.030930] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.031841] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.032650] Workqueue: events register_cache_worker [bcache]
[ 4.033149] Call Trace:
[ 4.033549] <TASK>
[ 4.033972] dump_stack_lvl+0x48/0x70
[ 4.034418] dump_stack+0x10/0x20
[ 4.034839] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.035279] btree_mergesort+0x4d4/0x520 [bcache]
[ 4.035730] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.036191] ? __pfx_bch_extent_sort_cmp+0x10/0x10 [bcache]
[ 4.036691] __btree_sort+0x96/0x2d0 [bcache]
[ 4.037182] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.037674] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.038172] ? mempool_kfree+0xe/0x20
[ 4.038617] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.039120] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.039659] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.040220] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.040806] ? __pfx_up_write+0x10/0x10
[ 4.041371] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.041921] run_cache_set+0x596/0x840 [bcache]
[ 4.042497] register_cache_set+0x1a2/0x210 [bcache]
[ 4.043089] register_cache+0x11a/0x1a0 [bcache]
[ 4.043715] register_cache_worker+0x22/0x80 [bcache]
[ 4.044348] process_one_work+0x23d/0x450
[ 4.044887] worker_thread+0x50/0x3f0
[ 4.045422] ? __pfx_worker_thread+0x10/0x10
[ 4.045936] kthread+0xef/0x120
[ 4.046445] ? __pfx_kthread+0x10/0x10
[ 4.046942] ret_from_fork+0x44/0x70
[ 4.047423] ? __pfx_kthread+0x10/0x10
[ 4.047878] ret_from_fork_asm+0x1b/0x30
[ 4.048339] </TASK>
[ 4.227653] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:281:4
[ 4.228847] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.229472] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.230680] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.231954] Workqueue: events register_cache_worker [bcache]
[ 4.232690] Call Trace:
[ 4.233327] <TASK>
[ 4.233935] dump_stack_lvl+0x48/0x70
[ 4.234568] dump_stack+0x10/0x20
[ 4.235219] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.235833] bch_extent_sort_fixup+0xb95/0xd70 [bcache]
[ 4.236524] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.237159] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.237839] btree_mergesort+0x221/0x520 [bcache]
[ 4.238823] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.239800] __btree_sort+0x96/0x2d0 [bcache]
[ 4.240880] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.243046] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.245223] ? mempool_kfree+0xe/0x20
[ 4.246311] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.247410] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.248471] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.248959] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.249454] ? __pfx_up_write+0x10/0x10
[ 4.249904] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.250386] run_cache_set+0x596/0x840 [bcache]
[ 4.250842] register_cache_set+0x1a2/0x210 [bcache]
[ 4.251319] register_cache+0x11a/0x1a0 [bcache]
[ 4.251748] register_cache_worker+0x22/0x80 [bcache]
[ 4.252181] process_one_work+0x23d/0x450
[ 4.252559] worker_thread+0x50/0x3f0
[ 4.252922] ? __pfx_worker_thread+0x10/0x10
[ 4.253286] kthread+0xef/0x120
[ 4.253659] ? __pfx_kthread+0x10/0x10
[ 4.254024] ret_from_fork+0x44/0x70
[ 4.254394] ? __pfx_kthread+0x10/0x10
[ 4.254755] ret_from_fork_asm+0x1b/0x30
[ 4.255145] </TASK>
[ 4.257388] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:36:18
[ 4.258429] index 14 is out of range for type 'btree_iter_set [4]'
[ 4.258964] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.260073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.261188] Workqueue: events register_cache_worker [bcache]
[ 4.261811] Call Trace:
[ 4.262374] <TASK>
[ 4.262912] dump_stack_lvl+0x48/0x70
[ 4.263502] dump_stack+0x10/0x20
[ 4.264042] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.264605] bch_extent_sort_fixup+0xbe5/0xd70 [bcache]
[ 4.265218] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.265821] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.266514] btree_mergesort+0x221/0x520 [bcache]
[ 4.267234] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.267882] __btree_sort+0x96/0x2d0 [bcache]
[ 4.268508] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.269144] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.269825] ? mempool_kfree+0xe/0x20
[ 4.270489] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.271243] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.272293] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.273260] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.274182] ? __pfx_up_write+0x10/0x10
[ 4.274973] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.276053] run_cache_set+0x596/0x840 [bcache]
[ 4.276972] register_cache_set+0x1a2/0x210 [bcache]
[ 4.277865] register_cache+0x11a/0x1a0 [bcache]
[ 4.278703] register_cache_worker+0x22/0x80 [bcache]
[ 4.279907] process_one_work+0x23d/0x450
[ 4.280690] worker_thread+0x50/0x3f0
[ 4.282228] ? __pfx_worker_thread+0x10/0x10
[ 4.283082] kthread+0xef/0x120
[ 4.283467] ? __pfx_kthread+0x10/0x10
[ 4.283803] ret_from_fork+0x44/0x70
[ 4.284143] ? __pfx_kthread+0x10/0x10
[ 4.284474] ret_from_fork_asm+0x1b/0x30
[ 4.284807] </TASK>
[ 4.286129] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:291:4
[ 4.286791] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.287231] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.288033] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.288863] Workqueue: events register_cache_worker [bcache]
[ 4.289340] Call Trace:
[ 4.289753] <TASK>
[ 4.290168] dump_stack_lvl+0x48/0x70
[ 4.290581] dump_stack+0x10/0x20
[ 4.290984] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.291432] bch_extent_sort_fixup+0xb77/0xd70 [bcache]
[ 4.291882] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.292309] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.292764] btree_mergesort+0x221/0x520 [bcache]
[ 4.293225] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.293683] __btree_sort+0x96/0x2d0 [bcache]
[ 4.294153] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.294631] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.295175] ? mempool_kfree+0xe/0x20
[ 4.295671] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.296257] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.296834] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.297446] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.298087] ? __pfx_up_write+0x10/0x10
[ 4.298678] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.299336] run_cache_set+0x596/0x840 [bcache]
[ 4.299941] register_cache_set+0x1a2/0x210 [bcache]
[ 4.300556] register_cache+0x11a/0x1a0 [bcache]
[ 4.301257] register_cache_worker+0x22/0x80 [bcache]
[ 4.302031] process_one_work+0x23d/0x450
[ 4.302722] worker_thread+0x50/0x3f0
[ 4.303410] ? __pfx_worker_thread+0x10/0x10
[ 4.304008] kthread+0xef/0x120
[ 4.304529] ? __pfx_kthread+0x10/0x10
[ 4.304910] ret_from_fork+0x44/0x70
[ 4.305315] ? __pfx_kthread+0x10/0x10
[ 4.305690] ret_from_fork_asm+0x1b/0x30
[ 4.306037] </TASK>
[Where problems could occur]
-The patch modifies the way bcache allocates space to the btree iterator. The main problems that could occur are different UBSAN warnings showing up that could possibly trigger a crash much easier than the current array index-out-of-bounds being observed.
Thank you @illwieckz for the original bug report
[original description]
Since I upgraded from lunar to mantic I get a load of those errors (41 on a fresh boot) in dmesg:
```
[ 4.277343] UBSAN: array-index-out-of-bounds in /build/linux-D15vQj/linux-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 4.277728] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.277925] CPU: 7 PID: 247 Comm: kworker/7:1 Not tainted 6.5.0-9-generic #9-Ubuntu
[ 4.278132] Hardware name: Default string Default string/Default string, BIOS WRX80SU8-F6 06/08/2023
[ 4.278531] Workqueue: events register_cache_worker [bcache]
[ 4.278754] Call Trace:
[ 4.278949] <TASK>
[ 4.279143] dump_stack_lvl+0x48/0x70
[ 4.279337] dump_stack+0x10/0x20
[ 4.279526] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.279721] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 4.279929] bch_btree_node_read_done+0xcb/0x410 [bcache]
[ 4.280142] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.280349] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.280557] bch_btree_node_get.part.0+0x15c/0x330 [bcache]
[ 4.280764] ? __bch_btree_ptr_invalid+0x66/0xe0 [bcache]
[ 4.280975] ? __pfx_up_write+0x10/0x10
[ 4.281170] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.281375] run_cache_set+0x596/0x850 [bcache]
[ 4.281578] ? srso_return_thunk+0x5/0x10
[ 4.281773] register_cache_set+0x1a2/0x210 [bcache]
[ 4.281984] register_cache+0x11a/0x1a0 [bcache]
[ 4.282187] register_cache_worker+0x22/0x80 [bcache]
[ 4.282387] process_one_work+0x223/0x440
[ 4.282573] worker_thread+0x4d/0x3f0
[ 4.282753] ? srso_return_thunk+0x5/0x10
[ 4.282931] ? _raw_spin_lock_irqsave+0xe/0x20
[ 4.283113] ? __pfx_worker_thread+0x10/0x10
[ 4.283286] kthread+0xf2/0x120
[ 4.283458] ? __pfx_kthread+0x10/0x10
[ 4.283631] ret_from_fork+0x47/0x70
[ 4.283800] ? __pfx_kthread+0x10/0x10
[ 4.283972] ret_from_fork_asm+0x1b/0x30
[ 4.284143] </TASK>
```
This system has 4 bcache backing devices and 4 bcache cache devices, though they are not associated for now and caching is disabled. It was already like that when I upgraded, so the kernel only uses the backing code, not the caching one.
ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: linux-image-6.5.0-9-generic 6.5.0-9.9
ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
Uname: Linux 6.5.0-9-generic x86_64
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: GNOME
Date: Sat Oct 14 23:16:33 2023
HibernationDevice: RESUME=none
MachineType: {report['dmi.sys.vendor']} {report['dmi.product.name']}
ProcFB:
0 amdgpudrmfb
1 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/@/boot/vmlinuz-6.5.0-9-generic root=UUID=f35ecf77-511e-4dde-ac11-c1d848e97315 ro rootflags=subvol=@ amdgpu.si_support=1 radeon.si_support=0 amdgpu.cik_support=1 radeon.cik_support=0 amdgpu.exp_hw_support=1 amdgpu.gpu_recovery=1 amdgpu.ppfeaturemask=0xffffffff delayacct zswap.enabled=1
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
linux-restricted-modules-6.5.0-9-generic N/A
linux-backports-modules-6.5.0-9-generic N/A
linux-firmware 20230919.git3672ccab-0ubuntu2.1
RfKill:
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/08/2023
dmi.bios.release: 5.23
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: WRX80SU8-F6
dmi.board.asset.tag: Default string
dmi.board.name: Default string
dmi.board.vendor: Default string
dmi.board.version: Default string
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInternational,LLC.:bvrWRX80SU8-F6:bd06/08/2023:br5.23:svnDefaultstring:pnDefaultstring:pvrDefaultstring:rvnDefaultstring:rnDefaultstring:rvrDefaultstring:cvnDefaultstring:ct3:cvrDefaultstring:skuDefaultstring:
dmi.product.family: Default string
dmi.product.name: Default string
dmi.product.sku: Default string
dmi.product.version: Default string
dmi.sys.vendor: Default string
modified.conffile..etc.default.apport: [modified]
mtime.conffile..etc.default.apport: 2018-06-16T17:39:00.798346 |
|
| 2024-07-01 20:59:31 |
Ghadi Rahme |
description |
[Impact]
Currently there are UBSAN warnings that show up when running bcache on jammy HWE, Mantic and noble. For now no side effects have been observed but such an issue could potentially cause a crash or corrupt data.
[Fix]
There is currently a fix upstream provided by the following patch:
* 3a861560ccb3 "bcache: fix variable length array abuse in btree_iter"
[Test Case]
1. Setup bcache on a jammy HWE kernel or mantic or noble machine. This can be done following the steps in this wiki: https://wiki.ubuntu.com/ServerTeam/Bcache
2. Restart the machine
3. After restarting the machine, the following UBSAN warnings and call traces can be seen in dmesg:
[ 3.824281] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 3.826338] index 4 is out of range for type 'btree_iter_set [4]'
[ 3.826812] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 3.827817] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 3.828835] Workqueue: events register_cache_worker [bcache]
[ 3.829429] Call Trace:
[ 3.830626] <TASK>
[ 3.831638] dump_stack_lvl+0x48/0x70
[ 3.832227] dump_stack+0x10/0x20
[ 3.832785] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 3.833357] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 3.834052] bch_btree_node_read_done+0xfc/0x450 [bcache]
[ 3.834653] ? mempool_kfree+0xe/0x20
[ 3.835211] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 3.835832] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 3.836474] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 3.837161] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 3.837838] ? __pfx_up_write+0x10/0x10
[ 3.838739] bch_btree_node_get+0x16/0x30 [bcache]
[ 3.839506] run_cache_set+0x596/0x840 [bcache]
[ 3.840197] register_cache_set+0x1a2/0x210 [bcache]
[ 3.840748] register_cache+0x11a/0x1a0 [bcache]
[ 3.841303] register_cache_worker+0x22/0x80 [bcache]
[ 3.841840] process_one_work+0x23d/0x450
[ 3.842297] worker_thread+0x50/0x3f0
[ 3.842698] ? __pfx_worker_thread+0x10/0x10
[ 3.843081] kthread+0xef/0x120
[ 3.843521] ? __pfx_kthread+0x10/0x10
[ 3.843892] ret_from_fork+0x44/0x70
[ 3.844264] ? __pfx_kthread+0x10/0x10
[ 3.844611] ret_from_fork_asm+0x1b/0x30
[ 3.844949] </TASK>
[ 4.029242] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1207:3
[ 4.030496] index 14 is out of range for type 'btree_iter_set [4]'
[ 4.030930] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.031841] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.032650] Workqueue: events register_cache_worker [bcache]
[ 4.033149] Call Trace:
[ 4.033549] <TASK>
[ 4.033972] dump_stack_lvl+0x48/0x70
[ 4.034418] dump_stack+0x10/0x20
[ 4.034839] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.035279] btree_mergesort+0x4d4/0x520 [bcache]
[ 4.035730] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.036191] ? __pfx_bch_extent_sort_cmp+0x10/0x10 [bcache]
[ 4.036691] __btree_sort+0x96/0x2d0 [bcache]
[ 4.037182] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.037674] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.038172] ? mempool_kfree+0xe/0x20
[ 4.038617] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.039120] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.039659] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.040220] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.040806] ? __pfx_up_write+0x10/0x10
[ 4.041371] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.041921] run_cache_set+0x596/0x840 [bcache]
[ 4.042497] register_cache_set+0x1a2/0x210 [bcache]
[ 4.043089] register_cache+0x11a/0x1a0 [bcache]
[ 4.043715] register_cache_worker+0x22/0x80 [bcache]
[ 4.044348] process_one_work+0x23d/0x450
[ 4.044887] worker_thread+0x50/0x3f0
[ 4.045422] ? __pfx_worker_thread+0x10/0x10
[ 4.045936] kthread+0xef/0x120
[ 4.046445] ? __pfx_kthread+0x10/0x10
[ 4.046942] ret_from_fork+0x44/0x70
[ 4.047423] ? __pfx_kthread+0x10/0x10
[ 4.047878] ret_from_fork_asm+0x1b/0x30
[ 4.048339] </TASK>
[ 4.227653] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:281:4
[ 4.228847] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.229472] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.230680] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.231954] Workqueue: events register_cache_worker [bcache]
[ 4.232690] Call Trace:
[ 4.233327] <TASK>
[ 4.233935] dump_stack_lvl+0x48/0x70
[ 4.234568] dump_stack+0x10/0x20
[ 4.235219] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.235833] bch_extent_sort_fixup+0xb95/0xd70 [bcache]
[ 4.236524] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.237159] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.237839] btree_mergesort+0x221/0x520 [bcache]
[ 4.238823] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.239800] __btree_sort+0x96/0x2d0 [bcache]
[ 4.240880] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.243046] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.245223] ? mempool_kfree+0xe/0x20
[ 4.246311] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.247410] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.248471] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.248959] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.249454] ? __pfx_up_write+0x10/0x10
[ 4.249904] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.250386] run_cache_set+0x596/0x840 [bcache]
[ 4.250842] register_cache_set+0x1a2/0x210 [bcache]
[ 4.251319] register_cache+0x11a/0x1a0 [bcache]
[ 4.251748] register_cache_worker+0x22/0x80 [bcache]
[ 4.252181] process_one_work+0x23d/0x450
[ 4.252559] worker_thread+0x50/0x3f0
[ 4.252922] ? __pfx_worker_thread+0x10/0x10
[ 4.253286] kthread+0xef/0x120
[ 4.253659] ? __pfx_kthread+0x10/0x10
[ 4.254024] ret_from_fork+0x44/0x70
[ 4.254394] ? __pfx_kthread+0x10/0x10
[ 4.254755] ret_from_fork_asm+0x1b/0x30
[ 4.255145] </TASK>
[ 4.257388] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:36:18
[ 4.258429] index 14 is out of range for type 'btree_iter_set [4]'
[ 4.258964] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.260073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.261188] Workqueue: events register_cache_worker [bcache]
[ 4.261811] Call Trace:
[ 4.262374] <TASK>
[ 4.262912] dump_stack_lvl+0x48/0x70
[ 4.263502] dump_stack+0x10/0x20
[ 4.264042] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.264605] bch_extent_sort_fixup+0xbe5/0xd70 [bcache]
[ 4.265218] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.265821] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.266514] btree_mergesort+0x221/0x520 [bcache]
[ 4.267234] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.267882] __btree_sort+0x96/0x2d0 [bcache]
[ 4.268508] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.269144] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.269825] ? mempool_kfree+0xe/0x20
[ 4.270489] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.271243] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.272293] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.273260] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.274182] ? __pfx_up_write+0x10/0x10
[ 4.274973] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.276053] run_cache_set+0x596/0x840 [bcache]
[ 4.276972] register_cache_set+0x1a2/0x210 [bcache]
[ 4.277865] register_cache+0x11a/0x1a0 [bcache]
[ 4.278703] register_cache_worker+0x22/0x80 [bcache]
[ 4.279907] process_one_work+0x23d/0x450
[ 4.280690] worker_thread+0x50/0x3f0
[ 4.282228] ? __pfx_worker_thread+0x10/0x10
[ 4.283082] kthread+0xef/0x120
[ 4.283467] ? __pfx_kthread+0x10/0x10
[ 4.283803] ret_from_fork+0x44/0x70
[ 4.284143] ? __pfx_kthread+0x10/0x10
[ 4.284474] ret_from_fork_asm+0x1b/0x30
[ 4.284807] </TASK>
[ 4.286129] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:291:4
[ 4.286791] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.287231] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.288033] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.288863] Workqueue: events register_cache_worker [bcache]
[ 4.289340] Call Trace:
[ 4.289753] <TASK>
[ 4.290168] dump_stack_lvl+0x48/0x70
[ 4.290581] dump_stack+0x10/0x20
[ 4.290984] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.291432] bch_extent_sort_fixup+0xb77/0xd70 [bcache]
[ 4.291882] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.292309] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.292764] btree_mergesort+0x221/0x520 [bcache]
[ 4.293225] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.293683] __btree_sort+0x96/0x2d0 [bcache]
[ 4.294153] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.294631] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.295175] ? mempool_kfree+0xe/0x20
[ 4.295671] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.296257] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.296834] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.297446] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.298087] ? __pfx_up_write+0x10/0x10
[ 4.298678] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.299336] run_cache_set+0x596/0x840 [bcache]
[ 4.299941] register_cache_set+0x1a2/0x210 [bcache]
[ 4.300556] register_cache+0x11a/0x1a0 [bcache]
[ 4.301257] register_cache_worker+0x22/0x80 [bcache]
[ 4.302031] process_one_work+0x23d/0x450
[ 4.302722] worker_thread+0x50/0x3f0
[ 4.303410] ? __pfx_worker_thread+0x10/0x10
[ 4.304008] kthread+0xef/0x120
[ 4.304529] ? __pfx_kthread+0x10/0x10
[ 4.304910] ret_from_fork+0x44/0x70
[ 4.305315] ? __pfx_kthread+0x10/0x10
[ 4.305690] ret_from_fork_asm+0x1b/0x30
[ 4.306037] </TASK>
[Where problems could occur]
-The patch modifies the way bcache allocates space to the btree iterator. The main problems that could occur are different UBSAN warnings showing up that could possibly trigger a crash much easier than the current array index-out-of-bounds being observed.
Thank you @illwieckz for the original bug report
[original description]
Since I upgraded from lunar to mantic I get a load of those errors (41 on a fresh boot) in dmesg:
```
[ 4.277343] UBSAN: array-index-out-of-bounds in /build/linux-D15vQj/linux-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 4.277728] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.277925] CPU: 7 PID: 247 Comm: kworker/7:1 Not tainted 6.5.0-9-generic #9-Ubuntu
[ 4.278132] Hardware name: Default string Default string/Default string, BIOS WRX80SU8-F6 06/08/2023
[ 4.278531] Workqueue: events register_cache_worker [bcache]
[ 4.278754] Call Trace:
[ 4.278949] <TASK>
[ 4.279143] dump_stack_lvl+0x48/0x70
[ 4.279337] dump_stack+0x10/0x20
[ 4.279526] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.279721] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 4.279929] bch_btree_node_read_done+0xcb/0x410 [bcache]
[ 4.280142] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.280349] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.280557] bch_btree_node_get.part.0+0x15c/0x330 [bcache]
[ 4.280764] ? __bch_btree_ptr_invalid+0x66/0xe0 [bcache]
[ 4.280975] ? __pfx_up_write+0x10/0x10
[ 4.281170] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.281375] run_cache_set+0x596/0x850 [bcache]
[ 4.281578] ? srso_return_thunk+0x5/0x10
[ 4.281773] register_cache_set+0x1a2/0x210 [bcache]
[ 4.281984] register_cache+0x11a/0x1a0 [bcache]
[ 4.282187] register_cache_worker+0x22/0x80 [bcache]
[ 4.282387] process_one_work+0x223/0x440
[ 4.282573] worker_thread+0x4d/0x3f0
[ 4.282753] ? srso_return_thunk+0x5/0x10
[ 4.282931] ? _raw_spin_lock_irqsave+0xe/0x20
[ 4.283113] ? __pfx_worker_thread+0x10/0x10
[ 4.283286] kthread+0xf2/0x120
[ 4.283458] ? __pfx_kthread+0x10/0x10
[ 4.283631] ret_from_fork+0x47/0x70
[ 4.283800] ? __pfx_kthread+0x10/0x10
[ 4.283972] ret_from_fork_asm+0x1b/0x30
[ 4.284143] </TASK>
```
This system has 4 bcache backing devices and 4 bcache cache devices, though they are not associated for now and caching is disabled. It was already like that when I upgraded, so the kernel only uses the backing code, not the caching one.
ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: linux-image-6.5.0-9-generic 6.5.0-9.9
ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
Uname: Linux 6.5.0-9-generic x86_64
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: GNOME
Date: Sat Oct 14 23:16:33 2023
HibernationDevice: RESUME=none
MachineType: {report['dmi.sys.vendor']} {report['dmi.product.name']}
ProcFB:
0 amdgpudrmfb
1 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/@/boot/vmlinuz-6.5.0-9-generic root=UUID=f35ecf77-511e-4dde-ac11-c1d848e97315 ro rootflags=subvol=@ amdgpu.si_support=1 radeon.si_support=0 amdgpu.cik_support=1 radeon.cik_support=0 amdgpu.exp_hw_support=1 amdgpu.gpu_recovery=1 amdgpu.ppfeaturemask=0xffffffff delayacct zswap.enabled=1
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
linux-restricted-modules-6.5.0-9-generic N/A
linux-backports-modules-6.5.0-9-generic N/A
linux-firmware 20230919.git3672ccab-0ubuntu2.1
RfKill:
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/08/2023
dmi.bios.release: 5.23
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: WRX80SU8-F6
dmi.board.asset.tag: Default string
dmi.board.name: Default string
dmi.board.vendor: Default string
dmi.board.version: Default string
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInternational,LLC.:bvrWRX80SU8-F6:bd06/08/2023:br5.23:svnDefaultstring:pnDefaultstring:pvrDefaultstring:rvnDefaultstring:rnDefaultstring:rvrDefaultstring:cvnDefaultstring:ct3:cvrDefaultstring:skuDefaultstring:
dmi.product.family: Default string
dmi.product.name: Default string
dmi.product.sku: Default string
dmi.product.version: Default string
dmi.sys.vendor: Default string
modified.conffile..etc.default.apport: [modified]
mtime.conffile..etc.default.apport: 2018-06-16T17:39:00.798346 |
[Impact]
Currently there are UBSAN warnings that show up when running bcache on jammy HWE, Mantic and noble. For now no side effects have been observed but such an issue could potentially cause a crash or corrupt data.
[Fix]
There is currently a fix upstream provided by the following patch:
* 3a861560ccb3 "bcache: fix variable length array abuse in btree_iter"
[Test Case]
1. Setup bcache on a jammy HWE kernel or mantic or noble machine. This can be done following the steps in this wiki: https://wiki.ubuntu.com/ServerTeam/Bcache
2. Restart the machine
3. After restarting the machine, the following UBSAN warnings and call traces can be seen in dmesg:
[ 3.824281] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 3.826338] index 4 is out of range for type 'btree_iter_set [4]'
[ 3.826812] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 3.827817] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 3.828835] Workqueue: events register_cache_worker [bcache]
[ 3.829429] Call Trace:
[ 3.830626] <TASK>
[ 3.831638] dump_stack_lvl+0x48/0x70
[ 3.832227] dump_stack+0x10/0x20
[ 3.832785] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 3.833357] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 3.834052] bch_btree_node_read_done+0xfc/0x450 [bcache]
[ 3.834653] ? mempool_kfree+0xe/0x20
[ 3.835211] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 3.835832] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 3.836474] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 3.837161] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 3.837838] ? __pfx_up_write+0x10/0x10
[ 3.838739] bch_btree_node_get+0x16/0x30 [bcache]
[ 3.844949] </TASK>
[ 4.029242] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1207:3
[ 4.030496] index 14 is out of range for type 'btree_iter_set [4]'
[ 4.030930] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.031841] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.032650] Workqueue: events register_cache_worker [bcache]
[ 4.033149] Call Trace:
[ 4.033549] <TASK>
[ 4.033972] dump_stack_lvl+0x48/0x70
[ 4.034418] dump_stack+0x10/0x20
[ 4.034839] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.035279] btree_mergesort+0x4d4/0x520 [bcache]
[ 4.035730] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.036191] ? __pfx_bch_extent_sort_cmp+0x10/0x10 [bcache]
[ 4.036691] __btree_sort+0x96/0x2d0 [bcache]
[ 4.037182] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.037674] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.038172] ? mempool_kfree+0xe/0x20
[ 4.038617] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.039120] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.039659] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.040220] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.040806] ? __pfx_up_write+0x10/0x10
[ 4.041371] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.048339] </TASK>
[ 4.227653] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:281:4
[ 4.228847] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.229472] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.230680] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.231954] Workqueue: events register_cache_worker [bcache]
[ 4.232690] Call Trace:
[ 4.233327] <TASK>
[ 4.233935] dump_stack_lvl+0x48/0x70
[ 4.234568] dump_stack+0x10/0x20
[ 4.235219] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.235833] bch_extent_sort_fixup+0xb95/0xd70 [bcache]
[ 4.236524] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.237159] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.237839] btree_mergesort+0x221/0x520 [bcache]
[ 4.238823] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.239800] __btree_sort+0x96/0x2d0 [bcache]
[ 4.240880] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.243046] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.245223] ? mempool_kfree+0xe/0x20
[ 4.246311] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.247410] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.248471] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.248959] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.249454] ? __pfx_up_write+0x10/0x10
[ 4.249904] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.255145] </TASK>
[ 4.257388] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:36:18
[ 4.258429] index 14 is out of range for type 'btree_iter_set [4]'
[ 4.258964] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.260073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.261188] Workqueue: events register_cache_worker [bcache]
[ 4.261811] Call Trace:
[ 4.262374] <TASK>
[ 4.262912] dump_stack_lvl+0x48/0x70
[ 4.263502] dump_stack+0x10/0x20
[ 4.264042] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.264605] bch_extent_sort_fixup+0xbe5/0xd70 [bcache]
[ 4.265218] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.265821] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.266514] btree_mergesort+0x221/0x520 [bcache]
[ 4.267234] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.267882] __btree_sort+0x96/0x2d0 [bcache]
[ 4.268508] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.269144] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.269825] ? mempool_kfree+0xe/0x20
[ 4.270489] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.271243] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.272293] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.273260] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.274182] ? __pfx_up_write+0x10/0x10
[ 4.274973] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.284807] </TASK>
[ 4.286129] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:291:4
[ 4.286791] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.287231] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.288033] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.288863] Workqueue: events register_cache_worker [bcache]
[ 4.289340] Call Trace:
[ 4.289753] <TASK>
[ 4.290168] dump_stack_lvl+0x48/0x70
[ 4.290581] dump_stack+0x10/0x20
[ 4.290984] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.291432] bch_extent_sort_fixup+0xb77/0xd70 [bcache]
[ 4.291882] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.292309] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.292764] btree_mergesort+0x221/0x520 [bcache]
[ 4.293225] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.293683] __btree_sort+0x96/0x2d0 [bcache]
[ 4.294153] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.294631] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.295175] ? mempool_kfree+0xe/0x20
[ 4.295671] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.296257] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.296834] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.297446] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.298087] ? __pfx_up_write+0x10/0x10
[ 4.298678] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.306037] </TASK>
[Where problems could occur]
-The patch modifies the way bcache allocates space to the btree iterator. The main problems that could occur are different UBSAN warnings showing up that could possibly trigger a crash much easier than the current array index-out-of-bounds being observed.
Thank you @illwieckz for the original bug report
[original description]
Since I upgraded from lunar to mantic I get a load of those errors (41 on a fresh boot) in dmesg:
```
[ 4.277343] UBSAN: array-index-out-of-bounds in /build/linux-D15vQj/linux-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 4.277728] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.277925] CPU: 7 PID: 247 Comm: kworker/7:1 Not tainted 6.5.0-9-generic #9-Ubuntu
[ 4.278132] Hardware name: Default string Default string/Default string, BIOS WRX80SU8-F6 06/08/2023
[ 4.278531] Workqueue: events register_cache_worker [bcache]
[ 4.278754] Call Trace:
[ 4.278949] <TASK>
[ 4.279143] dump_stack_lvl+0x48/0x70
[ 4.279337] dump_stack+0x10/0x20
[ 4.279526] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.279721] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 4.279929] bch_btree_node_read_done+0xcb/0x410 [bcache]
[ 4.280142] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.280349] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.280557] bch_btree_node_get.part.0+0x15c/0x330 [bcache]
[ 4.280764] ? __bch_btree_ptr_invalid+0x66/0xe0 [bcache]
[ 4.280975] ? __pfx_up_write+0x10/0x10
[ 4.281170] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.281375] run_cache_set+0x596/0x850 [bcache]
[ 4.281578] ? srso_return_thunk+0x5/0x10
[ 4.281773] register_cache_set+0x1a2/0x210 [bcache]
[ 4.281984] register_cache+0x11a/0x1a0 [bcache]
[ 4.282187] register_cache_worker+0x22/0x80 [bcache]
[ 4.282387] process_one_work+0x223/0x440
[ 4.282573] worker_thread+0x4d/0x3f0
[ 4.282753] ? srso_return_thunk+0x5/0x10
[ 4.282931] ? _raw_spin_lock_irqsave+0xe/0x20
[ 4.283113] ? __pfx_worker_thread+0x10/0x10
[ 4.283286] kthread+0xf2/0x120
[ 4.283458] ? __pfx_kthread+0x10/0x10
[ 4.283631] ret_from_fork+0x47/0x70
[ 4.283800] ? __pfx_kthread+0x10/0x10
[ 4.283972] ret_from_fork_asm+0x1b/0x30
[ 4.284143] </TASK>
```
This system has 4 bcache backing devices and 4 bcache cache devices, though they are not associated for now and caching is disabled. It was already like that when I upgraded, so the kernel only uses the backing code, not the caching one.
ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: linux-image-6.5.0-9-generic 6.5.0-9.9
ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
Uname: Linux 6.5.0-9-generic x86_64
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: GNOME
Date: Sat Oct 14 23:16:33 2023
HibernationDevice: RESUME=none
MachineType: {report['dmi.sys.vendor']} {report['dmi.product.name']}
ProcFB:
0 amdgpudrmfb
1 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/@/boot/vmlinuz-6.5.0-9-generic root=UUID=f35ecf77-511e-4dde-ac11-c1d848e97315 ro rootflags=subvol=@ amdgpu.si_support=1 radeon.si_support=0 amdgpu.cik_support=1 radeon.cik_support=0 amdgpu.exp_hw_support=1 amdgpu.gpu_recovery=1 amdgpu.ppfeaturemask=0xffffffff delayacct zswap.enabled=1
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
linux-restricted-modules-6.5.0-9-generic N/A
linux-backports-modules-6.5.0-9-generic N/A
linux-firmware 20230919.git3672ccab-0ubuntu2.1
RfKill:
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/08/2023
dmi.bios.release: 5.23
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: WRX80SU8-F6
dmi.board.asset.tag: Default string
dmi.board.name: Default string
dmi.board.vendor: Default string
dmi.board.version: Default string
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInternational,LLC.:bvrWRX80SU8-F6:bd06/08/2023:br5.23:svnDefaultstring:pnDefaultstring:pvrDefaultstring:rvnDefaultstring:rnDefaultstring:rvrDefaultstring:cvnDefaultstring:ct3:cvrDefaultstring:skuDefaultstring:
dmi.product.family: Default string
dmi.product.name: Default string
dmi.product.sku: Default string
dmi.product.version: Default string
dmi.sys.vendor: Default string
modified.conffile..etc.default.apport: [modified]
mtime.conffile..etc.default.apport: 2018-06-16T17:39:00.798346 |
|
| 2024-07-02 12:01:55 |
Stefan Bader |
linux (Ubuntu Jammy): status |
New |
Invalid |
|
| 2024-07-02 12:02:09 |
Stefan Bader |
linux (Ubuntu Noble): importance |
Undecided |
Medium |
|
| 2024-07-04 18:30:26 |
Stefan Bader |
linux (Ubuntu Noble): status |
Confirmed |
Fix Committed |
|
| 2024-07-11 19:44:38 |
Ubuntu Kernel Bot |
tags |
amd64 apport-bug kernel-flexible-array mantic |
amd64 apport-bug kernel-flexible-array kernel-spammed-noble-linux-v2 mantic verification-needed-noble-linux |
|
