GDS force mitigation re-enabled in 6.10 (and 6.11) causing crashes

[Tormod Volden]

The (supposedly unintended) re-enabling of GDS force migration in the Ubuntu 6.10 kernels causes the AVX instruction to be disabled on older CPUs which have no available microcode update. This causes various programs to crash due to the unconditional use of AVX in libgnutls.so, libxul.so, etc.

Typically "traps" of "invalid opcode" will be seen in dmesg output along with the initial notice:

[ 0.121833] GDS: Microcode update needed! Disabling AVX as mitigation.
[ 0.121835] GDS: Mitigation: AVX disabled, no microcode

When GDS force mitigation appeared in the kernel, with default "y", it created a lot of issues like these and Ubuntu quickly patched all their kernels, this from the 6.2.0-28.29_6.2.0-31.31 diff:

==========

```
diff -u linux-6.2.0/debian.master/changelog linux-6.2.0/debian.master/changelog
--- linux-6.2.0/debian.master/changelog
+++ linux-6.2.0/debian.master/changelog
@@ -1,3 +1,13 @@
+linux (6.2.0-31.31) lunar; urgency=medium
+
+ * lunar/linux: 6.2.0-31.31 -proposed tracker (LP: #2031146)
+
+ * libgnutls report "trap invalid opcode" when trying to install packages over
+ https (LP: #2031093)
+ - [Config]: disable CONFIG_GDS_FORCE_MITIGATION
+
+ -- Thadeu Lima de Souza Cascardo <email address hidden> Mon, 14 Aug 2023 08:29:52 -0300
+
 linux (6.2.0-28.29) lunar; urgency=medium

   * lunar/linux: 6.2.0-28.29 -proposed tracker (LP: #2030547)
diff -u linux-6.2.0/debian.master/config/annotations linux-6.2.0/debian.master/config/annotations
--- linux-6.2.0/debian.master/config/annotations
+++ linux-6.2.0/debian.master/config/annotations
@@ -4992,7 +4992,7 @@
 CONFIG_GCC_VERSION policy<{'amd64': '120200', 'arm64': '120200', 'armhf': '120200', 'ppc64el': '120200', 'riscv64': '120200', 's390x': '120200'}>
 CONFIG_GCOV_KERNEL policy<{'amd64': 'n', 'arm64': 'n', 'armhf': 'n', 'ppc64el': 'n', 'riscv64': 'n', 's390x': 'n'}>
 CONFIG_GDB_SCRIPTS policy<{'amd64': 'y', 'arm64': 'y', 'armhf': 'y', 'ppc64el': 'y', 'riscv64': 'y', 's390x': 'y'}>
-CONFIG_GDS_FORCE_MITIGATION policy<{'amd64': 'y'}>
+CONFIG_GDS_FORCE_MITIGATION policy<{'amd64': 'n'}>
 CONFIG_GEMINI_ETHERNET policy<{'arm64': 'm', 'armhf': 'm', 'ppc64el': 'm', 'riscv64': 'm'}>
 CONFIG_GENERIC_ADC_BATTERY policy<{'amd64': 'm', 'arm64': 'm', 'armhf': 'm', 'ppc64el': 'm', 'riscv64': 'm'}>
 CONFIG_GENERIC_ADC_THERMAL policy<{'amd64': 'm', 'arm64': 'm', 'armhf': 'm', 'ppc64el': 'm', 'riscv64': 'm'}>
```

==========

In upstream 6.9 the option was renamed from CONFIG_GDS_FORCE_MITIGATION to CONFIG_MITIGATION_GDS_FORCE, but when Ubuntu jumped from 6.8 to 6.10, this customization was lost, seen in the 6.8.0-31.31_6.10.0-15.15 diff:

==========

 ```
 CONFIG_GDB_SCRIPTS policy<{'amd64': 'y', 'arm64': 'y', 'armhf': 'y', 'ppc64el': 'y', 'riscv64': 'y', 's390x': 'y'}>
-CONFIG_GDS_FORCE_MITIGATION policy<{'amd64': 'n'}>
 CONFIG_GEMINI_ETHERNET policy<{'arm64': 'm', 'armhf': 'm', 'ppc64el': 'm', 'riscv64': 'm'}>
...
 CONFIG_MITIGATE_SPECTRE_BRANCH_HISTORY policy<{'arm64': 'y'}>
+CONFIG_MITIGATION_CALL_DEPTH_TRACKING policy<{'amd64': 'y'}>
+CONFIG_MITIGATION_GDS_FORCE policy<{'amd64': 'y'}>
+CONFIG_MITIGATION_IBPB_ENTRY policy<{'amd64': 'y'}>
+CONFIG_MITIGATION_IBRS_ENTRY policy<{'amd64': 'y'}>
```

==========

I am sure this was an oversight, and that the old option was simply dropped because it didn't exist any longer, without thinking of it being renamed (among a lot of other renames).

[Philip Cox (philcox)]

Hello Tormod, thank you for taking the time to open this bug report.

I just checked the latest 6.10 ubuntu kernel tree, and this seems to be resolved when the tree rebased to 6.10rc1.

Here is the current state of the kernel config:

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/oracular/tree/debian.master/config/annotations?h=Ubuntu-6.10.0-20.20

and on line 8134 you can see the expected config option.

And this is the change that added it:

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/oracular/commit/?h=master-next&id=05a2ea830dd81bb94bc7bd22755a90d685b4d4bf

[Tormod Volden (tormodvolden)]

Hi Philip,

There must be a misunderstanding. In your links I see:

+CONFIG_MITIGATION_GDS_FORCE policy<{'amd64': 'y'}>

So line 8134 has:
CONFIG_MITIGATION_GDS_FORCE policy<{'amd64': 'y'}>

and this is the problem. This setting should be 'n'.

Please see my original post. Thadeu changed this to 'n' (when it was still called CONFIG_GDS_FORCE_MITIGATION), to fix LP: #2031093. However, in 6.8.0-31.31_6.10.0-15.15 the option was renamed, and it was unintentionally (I believe) set to 'y' again.

[Tormod Volden (tormodvolden)]

Maybe I should point out that the upstream default is 'n' [1]. I don't know why it has been set to 'y' in Ubuntu, maybe just because mitigation sounds "safe".

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/Kconfig?h=v6.10#n2615

[Matthew Ruffell (mruffell)]

Hi Tormod,

Could you check 6.11.0-4-generic for Oracular in this particular ppa?

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/unstable

The 6.10 kernel in -proposed will be replaced by this one, or a newer build eventually.

Just waiting for the kernel team to make the source available in the normal oracular git repo.

Thanks,
Matthew

[Tormod Volden (tormodvolden)]

Matthew, you can check this yourself. Just install the kernel and look for CONFIG_MITIGATION_GDS_FORCE in /boot/config-*. You don't need to install it either, you can just download the "modules" package that owns the config file and unpack it with dpkg-deb and grep the config file.

[Tormod Volden (tormodvolden)]

> Just waiting for the kernel team to make the source available in the normal oracular git repo.

It is here:
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/oracular/tree/debian.master/config/annotations?h=Ubuntu-6.11.0-4.4#n8189

Also -5.5 is affected and it on its way: https://bugs.launchpad.net/kernel-sru-workflow/+bug/2077435

Everybody running these kernels on an older Intel CPU (6th gen or older), or on newer ones but missing microcode updates, will have a Titanic experience :)

[Timo Aaltonen (tjaalton)]

applied, will be in -5.5 (the real one)