[Zesty/Artful] On ARM64 PCIE physical function passthrough guest fails to boot

Bug #1732804 reported by Manoj Iyer
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Critical
Manoj Iyer
Zesty
Won't Fix
Critical
Manoj Iyer
Artful
Fix Released
Critical
Manoj Iyer
Bionic
Fix Released
Critical
Manoj Iyer

Bug Description

[Impact]
Passing through a physical function like the Mellanox PCIE ethernet controller causes the guest to fail booting, and host reports Hardware Error.
== Host ==
[109920.834703] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4
[109920.842142] {1}[Hardware Error]: event severity: recoverable
[109920.847848] {1}[Hardware Error]: precise tstamp: 2017-11-16 23:20:05
[109920.854385] {1}[Hardware Error]: Error 0, type: recoverable
[109920.860111] {1}[Hardware Error]: section_type: PCIe error
[109920.865718] {1}[Hardware Error]: port_type: 0, PCIe end point
[109920.871708] {1}[Hardware Error]: version: 3.0
[109920.876343] {1}[Hardware Error]: command: 0x0006, status: 0x0010
[109920.882559] {1}[Hardware Error]: device_id: 0000:01:00.0
[109920.888113] {1}[Hardware Error]: slot: 0
[109920.892285] {1}[Hardware Error]: secondary_bus: 0x00
[109920.897489] {1}[Hardware Error]: vendor_id: 0x15b3, device_id: 0x1013
[109920.904172] {1}[Hardware Error]: class_code: 000002
[109920.909378] vfio-pci 0000:01:00.0: aer_status: 0x00040000, aer_mask: 0x00000000
[109920.916675] Malformed TLP
[109920.916678] vfio-pci 0000:01:00.0: aer_layer=Transaction Layer, aer_agent=Receiver ID
[109920.924573] vfio-pci 0000:01:00.0: aer_uncor_severity: 0x00062010
[109920.930736] vfio-pci 0000:01:00.0: TLP Header: 4a008040 00000100 01000000 00000000
[109920.938548] vfio-pci 0000:01:00.0: broadcast error_detected message
[109921.965056] pcieport 0000:00:00.0: downstream link has been reset
[109921.965062] vfio-pci 0000:01:00.0: broadcast mmio_enabled message
[109921.965066] vfio-pci 0000:01:00.0: broadcast resume message
[109921.965070] vfio-pci 0000:01:00.0: AER: Device recovery successful
== Guest ==
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[ 1.518252] kvm [1]: HYP mode not available
[ 2.578929] mlx5_core 0000:05:00.0: mlx5_core_set_issi:778:(pid 152): Failed to query ISSI err(-1) status(0) synd(0)
[ 2.582424] mlx5_core 0000:05:00.0: failed to set issi
[ 2.616756] mlx5_core 0000:05:00.0: mlx5_load_one failed with error code -1

This is because, virtualization of physical functions are broken on systems with Maximum Payload Size bigger than 128. QDF2400 FW tries to maximize this setting. We have observed an MPS of 512 on QDF2400 systems.

[Fix]
Patches are in linux-next:
523184972b28 vfio/pci: Virtualize Maximum Payload Size
cf0d53ba4947 vfio/pci: Virtualize Maximum Read Request Size

[Testing]
With the above patches applied the guest is able to boot when PCIE physical function is passthrough and we don't see the errors on the host system.
== On the Guest ==
ubuntu@ubuntu-pcitest:~$ lspci
00:00.0 Host bridge: Red Hat, Inc. Device 0008
00:01.0 PCI bridge: Red Hat, Inc. Device 000c
00:01.1 PCI bridge: Red Hat, Inc. Device 000c
00:01.2 PCI bridge: Red Hat, Inc. Device 000c
00:01.3 PCI bridge: Red Hat, Inc. Device 000c
00:01.4 PCI bridge: Red Hat, Inc. Device 000c
00:01.5 PCI bridge: Red Hat, Inc. Device 000c
01:00.0 Ethernet controller: Red Hat, Inc Virtio network device (rev 01)
02:00.0 Communication controller: Red Hat, Inc Virtio console (rev 01)
03:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01)
04:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01)
05:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

ubuntu@ubuntu-pcitest:~$ lsmod | grep mlx
mlx5_core 471040 0
devlink 36864 1 mlx5_core
ptp 28672 1 mlx5_core

[Regression Potential]
Two patches to drivers/vfio/pci were cleanly cherry picked from linux-next and applied to Artful/Zesty. Tested on ARM64 QDF2400 system and no regressions were found.

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1732804

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: artful
Revision history for this message
Manoj Iyer (manjo) wrote :
Revision history for this message
Sinan Kaya (okaya) wrote :

I confirm that the test kernel fixes the issue. I was able to assign intel e1000e card to the guest machine and shutdown properly without observing any errors on the host OS.

guest lspci

[root@localhost ~]# lspci
00:00.0 Host bridge: Red Hat, Inc. Device 0008
00:01.0 PCI bridge: Red Hat, Inc. Device 000c
00:01.1 PCI bridge: Red Hat, Inc. Device 000c
00:01.2 PCI bridge: Red Hat, Inc. Device 000c
01:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
02:00.0 SCSI storage controller: Red Hat, Inc Virtio SCSI (rev 01)
[root@localhost ~]#

host lspci
ubuntu@ubuntu:~$ uname -a
Linux ubuntu 4.13.0-17-generic #20~lp1732804+build.1-Ubuntu SMP Fri Nov 17 00:45:36 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux

ubuntu@ubuntu:~$ lspci
0001:00:00.0 PCI bridge: Qualcomm Datacenter Technologies QDF2400 PCI Express Root Port
0002:00:00.0 PCI bridge: Qualcomm Datacenter Technologies QDF2400 PCI Express Root Port
0002:01:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
0004:00:00.0 PCI bridge: Qualcomm Datacenter Technologies QDF2400 PCI Express Root Port
0005:00:00.0 PCI bridge: Qualcomm Datacenter Technologies QDF2400 PCI Express Root Port

Manoj Iyer (manjo)
description: updated
description: updated
Manoj Iyer (manjo)
Changed in linux (Ubuntu):
assignee: Manoj Iyer (manjo) → Canonical Kernel Team (canonical-kernel-team)
status: Incomplete → In Progress
Changed in linux (Ubuntu Artful):
status: New → In Progress
Changed in linux (Ubuntu Zesty):
status: New → In Progress
Changed in linux (Ubuntu Artful):
importance: Undecided → Critical
Changed in linux (Ubuntu Zesty):
importance: Undecided → Critical
Changed in linux (Ubuntu Bionic):
assignee: Canonical Kernel Team (canonical-kernel-team) → Manoj Iyer (manjo)
Changed in linux (Ubuntu Artful):
assignee: nobody → Manoj Iyer (manjo)
Changed in linux (Ubuntu Zesty):
assignee: nobody → Manoj Iyer (manjo)
Seth Forshee (sforshee)
Changed in linux (Ubuntu Bionic):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Artful):
status: In Progress → Fix Committed
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-artful' to 'verification-done-artful'. If the problem still exists, change the tag 'verification-needed-artful' to 'verification-failed-artful'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-artful
Changed in linux (Ubuntu Zesty):
status: In Progress → Won't Fix
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

Setting Zesty status to 'Won't fix' since it has reached EOL and won't receive kernel updates anymore.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (20.1 KiB)

This bug was fixed in the package linux - 4.13.0-36.40

---------------
linux (4.13.0-36.40) artful; urgency=medium

  * linux: 4.13.0-36.40 -proposed tracker (LP: #1750010)

  * Rebuild without "CVE-2017-5754 ARM64 KPTI fixes" patch set

linux (4.13.0-35.39) artful; urgency=medium

  * linux: 4.13.0-35.39 -proposed tracker (LP: #1748743)

  * CVE-2017-5715 (Spectre v2 Intel)
    - Revert "UBUNTU: SAUCE: turn off IBPB when full retpoline is present"
    - SAUCE: turn off IBRS when full retpoline is present
    - [Packaging] retpoline files must be sorted
    - [Packaging] pull in retpoline files

linux (4.13.0-34.37) artful; urgency=medium

  * linux: 4.13.0-34.37 -proposed tracker (LP: #1748475)

  * libata: apply MAX_SEC_1024 to all LITEON EP1 series devices (LP: #1743053)
    - libata: apply MAX_SEC_1024 to all LITEON EP1 series devices

  * KVM patches for s390x to provide facility bits 81 (ppa15) and 82 (bpb)
    (LP: #1747090)
    - KVM: s390: wire up bpb feature

  * artful 4.13 i386 kernels crash after memory hotplug remove (LP: #1747069)
    - Revert "mm, memory_hotplug: do not associate hotadded memory to zones until
      online"

  * CVE-2017-5715 (Spectre v2 Intel)
    - x86/feature: Enable the x86 feature to control Speculation
    - x86/feature: Report presence of IBPB and IBRS control
    - x86/enter: MACROS to set/clear IBRS and set IBPB
    - x86/enter: Use IBRS on syscall and interrupts
    - x86/idle: Disable IBRS entering idle and enable it on wakeup
    - x86/idle: Disable IBRS when offlining cpu and re-enable on wakeup
    - x86/mm: Set IBPB upon context switch
    - x86/mm: Only set IBPB when the new thread cannot ptrace current thread
    - x86/entry: Stuff RSB for entry to kernel for non-SMEP platform
    - x86/kvm: add MSR_IA32_SPEC_CTRL and MSR_IA32_PRED_CMD to kvm
    - x86/kvm: Set IBPB when switching VM
    - x86/kvm: Toggle IBRS on VM entry and exit
    - x86/spec_ctrl: Add sysctl knobs to enable/disable SPEC_CTRL feature
    - x86/spec_ctrl: Add lock to serialize changes to ibrs and ibpb control
    - x86/cpu/AMD: Add speculative control support for AMD
    - x86/microcode: Extend post microcode reload to support IBPB feature
    - KVM: SVM: Do not intercept new speculative control MSRs
    - x86/svm: Set IBRS value on VM entry and exit
    - x86/svm: Set IBPB when running a different VCPU
    - KVM: x86: Add speculative control CPUID support for guests
    - SAUCE: turn off IBPB when full retpoline is present

  * Artful 4.13 fixes for tun (LP: #1748846)
    - tun: call dev_get_valid_name() before register_netdevice()
    - tun: allow positive return values on dev_get_valid_name() call
    - tun/tap: sanitize TUNSETSNDBUF input

  * boot failure on AMD Raven + WestonXT (LP: #1742759)
    - SAUCE: drm/amdgpu: add atpx quirk handling (v2)

linux (4.13.0-33.36) artful; urgency=low

  * linux: 4.13.0-33.36 -proposed tracker (LP: #1746903)

  [ Stefan Bader ]
  * starting VMs causing retpoline4 to reboot (LP: #1747507) // CVE-2017-5715
    (Spectre v2 retpoline)
    - x86/retpoline: Fill RSB on context switch for affected CPUs
    - x86/retpoline: Add LFENCE to the retpoline/RSB filling RSB macros
    - x86/retpol...

Changed in linux (Ubuntu Artful):
status: Fix Committed → Fix Released
Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
Brad Figg (brad-figg)
tags: added: cscc
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.