UBSAN: array-index-out-of-bounds (drivers/net/hyperv/netvsc.c)

[Juan Pedro Paredes]

HiperV VM network problems

[ 19.259297] ================================================================================
[ 19.259536] UBSAN: array-index-out-of-bounds in /build/linux-7dWMY3/linux-6.5.0/drivers/net/hyperv/netvsc.c:1445:41
[ 19.259715] index 1 is out of range for type 'vmtransfer_page_range [1]'
[ 19.259896] CPU: 1 PID: 1306 Comm: (udev-worker) Not tainted 6.5.0-7-generic #7-Ubuntu
[ 19.259898] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 04/06/2022
[ 19.259899] Call Trace:
[ 19.259901] <IRQ>
[ 19.259902] dump_stack_lvl+0x48/0x70
[ 19.259908] dump_stack+0x10/0x20
[ 19.259909] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 19.259912] netvsc_receive+0x437/0x490 [hv_netvsc]
[ 19.259917] netvsc_poll+0x176/0x4b0 [hv_netvsc]
[ 19.259921] __napi_poll+0x30/0x1f0
[ 19.259924] net_rx_action+0x181/0x2e0
[ 19.259925] __do_softirq+0xd6/0x346
[ 19.259927] ? _raw_spin_unlock+0xe/0x40
[ 19.259929] __irq_exit_rcu+0x75/0xa0
[ 19.259932] irq_exit_rcu+0xe/0x20
[ 19.259933] sysvec_hyperv_callback+0x92/0xd0
[ 19.259935] </IRQ>
[ 19.259935] <TASK>

ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: ubuntu-release-upgrader-core 1:23.10.8
ProcVersionSignature: Ubuntu 6.5.0-7.7-generic 6.5.3
Uname: Linux 6.5.0-7-generic x86_64
ApportVersion: 2.27.0-0ubuntu4
Architecture: amd64
CasperMD5CheckResult: pass
CloudArchitecture: x86_64
CloudID: none
CloudName: none
CloudPlatform: none
CloudSubPlatform: config
CrashDB: ubuntu
Date: Sun Oct 8 23:09:45 2023
InstallationDate: Installed on 2021-03-07 (945 days ago)
InstallationMedia: Ubuntu-Server 20.04.2 LTS "Focal Fossa" - Release amd64 (20210201.2)
PackageArchitecture: all
ProcEnviron:
 LANG=es_ES.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm
 XDG_RUNTIME_DIR=<set>
SourcePackage: ubuntu-release-upgrader
Symptom: release-upgrade
UpgradeStatus: Upgraded to mantic on 2023-10-08 (0 days ago)
VarLogDistupgradeLspcitxt:

VarLogDistupgradeXorgFixuplog:
 INFO:root:/usr/bin/do-release-upgrade running
 INFO:root:No xorg.conf, exiting
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 oct 9 20:46 seq
 crw-rw---- 1 root audio 116, 33 oct 9 20:46 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: pass
CloudArchitecture: x86_64
CloudID: none
CloudName: none
CloudPlatform: none
CloudSubPlatform: config
DistroRelease: Ubuntu 23.10
InstallationDate: Installed on 2021-03-07 (948 days ago)
InstallationMedia: Ubuntu-Server 20.04.2 LTS "Focal Fossa" - Release amd64 (20210201.2)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lspci:

Lspci-vt:

Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:

Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: {report['dmi.sys.vendor']} {report['dmi.product.name']}
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 LANG=es_ES.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm
 XDG_RUNTIME_DIR=<set>
ProcFB: 0 hyperv_drmdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.5.0-9-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro mitigations=off iommu=pt
ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-6.5.0-9-generic N/A
 linux-backports-modules-6.5.0-9-generic N/A
 linux-firmware 20230919.git3672ccab-0ubuntu2.1
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: mantic
Uname: Linux 6.5.0-9-generic x86_64
UpgradeStatus: Upgraded to mantic on 2023-10-08 (2 days ago)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 04/06/2022
dmi.bios.release: 4.1
dmi.bios.vendor: Microsoft Corporation
dmi.bios.version: Hyper-V UEFI Release v4.1
dmi.board.asset.tag: None
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: Hyper-V UEFI Release v4.1
dmi.chassis.asset.tag: 4585-4991-9142-0880-5352-0701-05
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: Hyper-V UEFI Release v4.1
dmi.modalias: dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev4.1:bd04/06/2022:br4.1:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev4.1:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev4.1:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev4.1:skuNone:
dmi.product.family: Virtual Machine
dmi.product.name: Virtual Machine
dmi.product.sku: None
dmi.product.version: Hyper-V UEFI Release v4.1
dmi.sys.vendor: Microsoft Corporation
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 oct 11 08:43 seq
 crw-rw---- 1 root audio 116, 33 oct 11 08:43 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: pass
CloudArchitecture: x86_64
CloudID: none
CloudName: none
CloudPlatform: none
CloudSubPlatform: config
DistroRelease: Ubuntu 23.10
InstallationDate: Installed on 2021-03-07 (948 days ago)
InstallationMedia: Ubuntu-Server 20.04.2 LTS "Focal Fossa" - Release amd64 (20210201.2)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lspci:

Lspci-vt:

Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:

Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: {report['dmi.sys.vendor']} {report['dmi.product.name']}
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 LANG=es_ES.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm
 XDG_RUNTIME_DIR=<set>
ProcFB: 0 hyperv_drmdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.5.0-9-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro mitigations=off
ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-6.5.0-9-generic N/A
 linux-backports-modules-6.5.0-9-generic N/A
 linux-firmware 20230919.git3672ccab-0ubuntu2.1
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: mantic
Uname: Linux 6.5.0-9-generic x86_64
UpgradeStatus: Upgraded to mantic on 2023-10-08 (2 days ago)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 04/06/2022
dmi.bios.release: 4.1
dmi.bios.vendor: Microsoft Corporation
dmi.bios.version: Hyper-V UEFI Release v4.1
dmi.board.asset.tag: None
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: Hyper-V UEFI Release v4.1
dmi.chassis.asset.tag: 4585-4991-9142-0880-5352-0701-05
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: Hyper-V UEFI Release v4.1
dmi.modalias: dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev4.1:bd04/06/2022:br4.1:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev4.1:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev4.1:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev4.1:skuNone:
dmi.product.family: Virtual Machine
dmi.product.name: Virtual Machine
dmi.product.sku: None
dmi.product.version: Hyper-V UEFI Release v4.1
dmi.sys.vendor: Microsoft Corporation

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2038777

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Juan Pedro Paredes (juampe)]

apport information

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Mike Ferreira (mafoelffen)]

I have a user that is affected by this bug. I am asking him to post here with his info. I ask that you reopen this... For him = His is a VM in Hyper-V.

RE: https://ubuntuforums.org/showthread.php?t=2494675

From what I can see, there was a Kernel Bug that was patched for this error upstream: https://<email address hidden>/T/

I'm wondering if that patch applied to kernel 6.5.0-14? (Ubuntu 22.04.3)

If so or not, he is still getting this error. Maybe we could get this kick-started back to life, to get this investigated and resolved(?)

[Song Hyo Jin (xenoside)]

I have the same error in hyper-v.
It is ubuntu 22.04 and below is the corresponding error part of dmesg.
It doesn't really prevent me from using it.

[ 4.589667] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-ZV1Qen/linux-hwe-6.5-6.5.0/drivers/net/hyperv/netvsc.c:1445:41
[ 4.589951] index 1 is out of range for type 'vmtransfer_page_range [1]'
[ 4.590211] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.5.0-18-generic #18~22.04.1-Ubuntu
[ 4.590216] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
[ 4.590218] Call Trace:
[ 4.590220] <IRQ>
[ 4.590224] dump_stack_lvl+0x48/0x70
[ 4.590232] dump_stack+0x10/0x20
[ 4.590234] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.590239] netvsc_receive+0x42f/0x480 [hv_netvsc]
[ 4.590247] netvsc_poll+0x176/0x4a0 [hv_netvsc]
[ 4.590252] ? srso_alias_return_thunk+0x5/0x7f
[ 4.590257] __napi_poll+0x33/0x1f0
[ 4.590261] net_rx_action+0x181/0x2e0
[ 4.590263] ? srso_alias_return_thunk+0x5/0x7f
[ 4.590265] ? srso_alias_return_thunk+0x5/0x7f
[ 4.590269] __do_softirq+0xdc/0x349
[ 4.590271] ? _raw_spin_unlock+0xe/0x40
[ 4.590273] ? srso_alias_return_thunk+0x5/0x7f
[ 4.590277] __irq_exit_rcu+0x75/0xa0
[ 4.590281] irq_exit_rcu+0xe/0x20
[ 4.590284] sysvec_hyperv_callback+0x92/0xd0
[ 4.590288] </IRQ>
[ 4.590289] <TASK>
[ 4.590290] asm_sysvec_hyperv_callback+0x1b/0x20
[ 4.590294] RIP: 0010:pv_native_safe_halt+0xb/0x10
[ 4.590297] Code: 0b 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 eb 07 0f 00 2d a9 20 2f 00 fb f4 <e9> 70 9b 01 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 55
[ 4.590299] RSP: 0018:ffffffff92403e20 EFLAGS: 00000246
[ 4.590301] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 4.590303] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 4.590304] RBP: ffffffff92403e28 R08: 0000000000000000 R09: 0000000000000000
[ 4.590305] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff9240fbc0
[ 4.590306] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 4.590310] ? default_idle+0x9/0x30
[ 4.590313] arch_cpu_idle+0x9/0x10
[ 4.590316] default_idle_call+0x2c/0xf0
[ 4.590318] cpuidle_idle_call+0x153/0x190
[ 4.590322] do_idle+0x82/0xf0
[ 4.590324] cpu_startup_entry+0x2a/0x30
[ 4.590326] rest_init+0xde/0x100
[ 4.590328] ? acpi_enable_subsystem+0xe6/0x2a0
[ 4.590331] ? static_key_disable+0x1f/0x30
[ 4.590336] arch_call_rest_init+0xe/0x30
[ 4.590339] start_kernel+0x34f/0x440
[ 4.590342] x86_64_start_reservations+0x18/0x30
[ 4.590346] x86_64_start_kernel+0xa4/0xe0
[ 4.590349] secondary_startup_64_no_verify+0x17e/0x18b
[ 4.590355] </TASK>
[ 4.590365] ================================================================================
[ 4.590631] ================================================================================
[ 4.590947] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-ZV1Qen/linux-hwe-6.5-6.5.0/drivers/net/hyperv/netvsc.c:1446:41
[ 4.591246] index 1 is out of range fo...

[Leonardo Zambrano (leonardozambrano)]

Hi there,

Since package linux-image-azure now install linux-image-6.5.0-1015-azure I suppose you will receive more reports about this.

After inspecting linux-6.5/include/linux/hyperv.h file from linux-image-unsigned-6.5.0-1015-azure source package, I confirmed it does not have patch "hv: hyperv.h: Replace one-element array with flexible-array member" [1] applied since among others, that one was actually just picked up for upstream 6.6 as per commit "[GIT PULL] Hyper-V commits for 6.6" [2].

Hopefully the relevant commit [1] can be applied to Ubuntu 6.5 kernel soon.

[1] https://<email address hidden>/
[2] https://lore.kernel.org/lkml/ZPQTFyfzgvlp3QkW@liuwe-devbox-debian-v2/

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Luca Boccassi (bluca)]

Github Actions enabled KVM for all open source repositories for free in January:

https://github.blog/2024-01-17-github-hosted-runners-double-the-power-for-open-source/

We started using it in systemd, and we hit this bug:

https://paste.centos.org/view/411107c8

This will start quickly affecting everybody who tries to use KVM in their CI jobs on Github. Could you please arrange for this fix to be backported to the azure kernel for jammy?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Leonardo Zambrano (leonardozambrano)]

Hi there,

I can confirm that the relevant fix/patch was backported to the azure kernel for jammy, linux-azure/(6.5.0.1024.25~22.04.1) kernel in jammy-updates solves the problem for Azure VMs.