[SRU][22.04.2 & 23.10] OS cannot boot successfully when enabling VMD in UEFI setup

[Adrian Huang]

[Impact]
When enabling VMD in UEFI setup, OS cannot boot successfully. And, the panic leads to the system reboot. The following log is shown:

[ 166.605518] DMAR: VT-d detected Invalidation Queue Error: Reason f
[ 166.605522] DMAR: VT-d detected Invalidation Time-out Error: SID ffff
[ 166.612445] DMAR: VT-d detected Invalidation Completion Error: SID ffff
[ 166.612447] DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
[ 166.612449] DMAR: QI PRIOR: UNKNOWN qw0 = 0x0, qw1 = 0x0
...

Additional info:
  * The issue happens on both Lenovo SE350 server and Lenovo SR850 v2 server.

Debugging info and fix commit info:
  * `git bisect` indicates the offending commit is 6aab5622296b ("PCI: vmd: Clean up domain before enumeration"). The root cause is that VMD driver tries to clear a PCI configuration space range when resetting a VMD domain (https://github.com/torvalds/linux/blob/master/drivers/pci/controller/vmd.c#L544), which leads to the failure.

[Fix]
  * Another `git bisect` indicates the fix commit is 20f3337d350c ("x86: don't use REP_GOOD or ERMS for small memory clearing). I confirmed that this commit can fix the issue.

Would it be possible to include the commit 20f3337d350c in Ubuntu 22.04.2/23.10 kernel?

[Test Plan]

Reproduce Step
1.Disable Intel VMD in BIOS settings
   System Settings --> Devices and I/O Ports --> Intel VMD technology --> Enable/Disable Intel VMD : Disabled

2.Install OS

3.Enable Intel VMD in BIOS settings
   System Settings --> Devices and I/O Ports --> Intel VMD technology --> Enable/Disable Intel VMD : Enabled

4.Rebooting will reproduce this issue

[ Where problems could occur ]
* Lenovo SE350 server and Lenovo SR850 v2 server
* The regression leads to the boot failure (cannot boot info OS successfully).

[ Other Info ]

https://code.launchpad.net/~mreed8855/ubuntu/+source/linux/+git/lunar/+ref/enable_vmd_lp_2020022

[Jeff Lane  (bladernr)]

Kernels seen 5.19, 6.2 so far.

Can you also try 5.15 (22.04 GA) and 5.4 (20.04 GA) as both of those are certified on both 22.04 and 20.04

[Jeff Lane  (bladernr)]

Adrian, two things:

Can you provide steps to recreate this using the SE350 (including whatever you're setting in BIOS) so I can see if I can provide a local sample with the failure?

Second:
20f3337d350c ("x86: don't use REP_GOOD or ERMS for small memory clearing)

Which kernel tree is this commit in? I could not find it in mainline (unless it has a different mainline commit ID from being merged).

[Adrian Huang (ahuang12)]

Jeff,

[Reproduce Step]
1.Disable Intel VMD in BIOS settings
   System Settings --> Devices and I/O Ports --> Intel VMD technology --> Enable/Disable Intel VMD : Disabled

2.Install OS

3.Enable Intel VMD in BIOS settings
   System Settings --> Devices and I/O Ports --> Intel VMD technology --> Enable/Disable Intel VMD : Enabled

4.Rebooting will reproduce this issue

[Commit 20f3337d350c]
This commit is from Linus's tree (merged in 6.4-rc1): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/arch/x86/lib/memset_64.S?id=20f3337d350c4e1b4ac66d731fd4e98565bf6cc0

[Jeff Lane  (bladernr)]

No need for Mantic, which already contains the patch. Only need to pull this back to Lunar.

[Michael Reed (mreed8855)]

I have created a test kernel, please test it and provide feedback.

https://people.canonical.com/~mreed/lenovo/lp_2020022_vmd/lunar/

[Michael Reed (mreed8855)]

Adrian,

Can you add to the "Where problems could occur" and provide the regression risk?

[Adrian Huang (ahuang12)]

The test kernel is still failed. Not sure if the patch is included correctly. Could you put the source deb package in your URL? I can check that.

[Jeff Lane  (bladernr)]

Can you install one of the daily ISOs for mantic (23.10) and test to see if this is an issue? The patches you mention are included in the Mantic kernel already, so that one should not see the failure.

We're still trying to figure out how to get you a working 6.2 kernel.

[Adrian Huang (ahuang12)]

Confirmed that the kernel (v6.5) of 23.10 does not have the issue.

[Michael Reed (mreed8855)]

I created a 6.2 test kernel for Lunar. Please test

https://people.canonical.com/~mreed/lenovo/lp_2020022_vmd/lunar/12062023/

[Adrian Huang (ahuang12)]

Confirmed that the test kernel in Comment #14 fixes the issue.

Thanks.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/6.2.0-41.42 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lunar-linux' to 'verification-done-lunar-linux'. If the problem still exists, change the tag 'verification-needed-lunar-linux' to 'verification-failed-lunar-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Brian Murray (brian-murray)]

Ubuntu 23.04 (Lunar Lobster) has reached end of life, so this bug will not be fixed for that specific release.