Azure - Kernel crashes when removing gpu from pci

[Ioanna Alifieraki]

[Description]

On a VM on Azure with a Tesla gpu it was noticed that when removing the gpu from the pci the vm would crash. In case the nvidia drivers are loaded, the machine won't crash. Instead the removing process will hang and the machine will crash on reboot.

This is related to bug [1].
The bug reported in [1] regards another driver but the root cause is the same.
It is still investigated whether this is a bug in pci, or it is a bug of various drivers on how they use pci.

For this case we have identified that removing commit [2] prevents the kernel crashes.

Azure has requested to revert this commit, at least for the time being.
This commit is not in upstream, so it just need to be reverted from Ubuntu kernels.

[Test Case]

On an Azure vm with a gpu :

# echo '1' > /sys/bus/pci/devices/0001:00:00.0/remove

where '0001:00:00.0' the pci address of the gpu.
The vm will crash.

[Where things could go wrong]

The commit to be reverted was included in a patchset to address lp bugs https://bugs.launchpad.net/bugs/2023071 and https://bugs.launchpad.net/bugs/2023594

However this commit just reduces boot time and removing shall not introduce any regressions.
Side effects will be increase in the boot time.

[Other]

Only Ubuntu azure kernels are affected :

- Jammy 5.15
- Lunar 6.2

Focal is also affected since it's using 5.15 kernel.
This commit does not appear in Mantic 6.5 kernel.

[1] https://bugzilla.kernel.org/show_bug.cgi?id=215515
[2] https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-azure/+git/jammy/commit/?h=Ubuntu-azure-5.15.0-1043.50&id=75af0c10b3703400890d314d1d91d25294234a81

[Ioanna Alifieraki (joalif)]

Upon further testing Lunar kernel 6.2 seems not be affected. I'll investigate further to find out why.

[Ioanna Alifieraki (joalif)]

It turns out that 6.2 is also affected but the bug appears a lot less often, once every ~20 reboots.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure/5.15.0-1055.63 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-azure' to 'verification-done-jammy-linux-azure'. If the problem still exists, change the tag 'verification-needed-jammy-linux-azure' to 'verification-failed-jammy-linux-azure'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ioanna Alifieraki (joalif)]

# VERIFICATION JAMMY

This verification is using a jammy 5.15.0-1055 kernel on a focal machine.
To reproduce this issue a special azure instance is required and currently I have access to a focal vm.

root@jo-twosla:/home/ubuntu# uname -rv
5.15.0-1055-azure #63~20.04.1-Ubuntu SMP Thu Jan 18 15:30:26 UTC 2024

root@jo-twosla:/home/ubuntu# lspci
0001:00:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
3240:00:02.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80)

root@jo-twosla:/home/ubuntu# echo '1' > /sys/bus/pci/devices/0001:00:00.0/remove

root@jo-twosla:/home/ubuntu# lspci
3240:00:02.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80)

The gpu is successfully removed without the vm crashing.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-azure - 5.15.0-1056.64

---------------
linux-azure (5.15.0-1056.64) jammy; urgency=medium

  * jammy/linux-azure: 5.15.0-1056.64 -proposed tracker (LP: #2052545)

  * Azure: Fix regression introduced in LP: #2045069 (LP: #2052453)
    - hv_netvsc: Fix race condition between netvsc_probe and netvsc_remove
    - hv_netvsc: Register VF in netvsc_probe if NET_DEVICE_REGISTER missed

linux-azure (5.15.0-1055.63) jammy; urgency=medium

  * jammy/linux-azure: 5.15.0-1055.63 -proposed tracker (LP: #2048291)

  * Azure - Kernel crashes when removing gpu from pci (LP: #2042568)
    - Revert "PCI: hv: Use async probing to reduce boot time"

  * Azure: mlx5e: Add support for PCI relaxed ordering (RO) for better
    performance (LP: #2039208)
    - RDMA/mlx5: Reorder calls to pcie_relaxed_ordering_enabled()
    - RDMA/mlx5: Remove pcie_relaxed_ordering_enabled() check for RO write

  * Azure: Deprecate Netvsc and implement MANA direct (LP: #2045069)
    - hv_netvsc: fix race of netvsc and VF register_netdevice
    - hv_netvsc: Fix race of register_netdevice_notifier and VF register
    - hv_netvsc: Mark VF as slave before exposing it to user-mode

  [ Ubuntu: 5.15.0-94.104 ]

  * jammy/linux: 5.15.0-94.104 -proposed tracker (LP: #2048777)
  * [SRU] Duplicate Device_dax ids Created and hence Probing is Failing.
    (LP: #2028158)
    - device-dax: Fix duplicate 'hmem' device registration
  * Add ODM driver f81604 usb-can (LP: #2045387)
    - can: usb: f81604: add Fintek F81604 support
    - [Config] updateconfigs for ODM drivers CONFIG_CAN_F81604
  * Add ODM driver gpio-m058ssan (LP: #2045386)
    - SAUCE: ODM: gpio: add M058SSAN gpio driver
    - [Config] updateconfigs for ODM drivers CONFIG_GPIO_M058SSAN
  * Add ODM driver rtc-pcf85263 (LP: #2045385)
    - SAUCE: ODM: rtc: add PCF85263 RTC driver
    - [Config] updateconfigs for ODM drivers CONFIG_RTC_DRV_PCF85263
  * AppArmor patch for mq-posix interface is missing in jammy (LP: #2045384)
    - SAUCE: (no-up) apparmor: reserve mediation classes
    - SAUCE: (no-up) apparmor: Add fine grained mediation of posix mqueues
  * Packaging resync (LP: #1786013)
    - [Packaging] update annotations scripts

  [ Ubuntu: 5.15.0-93.103 ]

  * jammy/linux: 5.15.0-93.103 -proposed tracker (LP: #2048330)
  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log
    - [Packaging] resync update-dkms-versions helper
    - [Packaging] remove helper scripts
    - [Packaging] update annotations scripts
    - debian/dkms-versions -- update from kernel-versions (main/2024.01.08)
  * Hotplugging SCSI disk in QEMU VM fails (LP: #2047382)
    - Revert "PCI: acpiphp: Reassign resources on bridge if necessary"
  * CVE-2023-6622
    - netfilter: nf_tables: bail out on mismatching dynset and set expressions
  * CVE-2024-0193
    - netfilter: nf_tables: skip set commit for deleted/destroyed sets
  * CVE-2023-6040
    - netfilter: nf_tables: Reject tables of unsupported family
  * Patches needed for AmpereOne (arm64) (LP: #2044192)
    - clocksource/arm_arch_timer: Add build-time guards for unhandled register
      accesses
    - clocksource/drivers/arm_arch_timer: Drop CNT*_T...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure-fips/5.15.0-1058.66+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-azure-fips' to 'verification-done-jammy-linux-azure-fips'. If the problem still exists, change the tag 'verification-needed-jammy-linux-azure-fips' to 'verification-failed-jammy-linux-azure-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!