severe performance issue after intel-microcode update

[Jim Simmons]

After upgrading my intel-microcode package to 3.20180807a.0ubuntu0.16.04.1 the performance for some test code that I have has worsened by 400-500%. Chip is an i7-3960X.

lsb_release -rd
Description: Ubuntu 16.04.5 LTS
Release: 16.04

apt-cache policy intel-microcode
intel-microcode:
  Installed: 3.20180807a.0ubuntu0.16.04.1
  Candidate: 3.20180807a.0ubuntu0.16.04.1
  Version table:
 *** 3.20180807a.0ubuntu0.16.04.1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status
     3.20151106.1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/restricted

[Henrique de Moraes Holschuh (hmh)]

I am afraid you need to disclose a _lot_ more information about your workload for us to even have an idea how you'd hit such an extreme slowdown effect...

Also, what mitigation modes have you enabled in the kernel, etc.

[Jim Simmons (jsimmons089)]

My workload is some trading performance tests that I've created, which

aren't  going to help you.  If you can tell me how to revert to a
previous intel-micrcode

package I can try and come up with some narrow isolated test cases where
the issue

occurs and send you those.

Thanks,

Jim

On 08/30/2018 05:50 AM, Henrique de Moraes Holschuh wrote:
> I am afraid you need to disclose a _lot_ more information about your
> workload for us to even have an idea how you'd hit such an extreme
> slowdown effect...
>
> Also, what mitigation modes have you enabled in the kernel, etc.
>

[Henrique de Moraes Holschuh (hmh)]

Basically, *you* need to know if:

1. your workload does a LOT of syscalls (kernel entries from userspace) OR

2. your workload does a LOT of VMEXITs (runs on a VM) OR

3. uses Intel SGX (you're on your own in that case, go talk to Intel).

Which are directly impacted by the microcode and/or kernel changes.

You can also selectively disable KPTI/PTI, L1D flushes (L1TF mitigation), and SSBD mitigation to track down which is not compatible with your workload. That *might* give enough hints on how to change your workload to not suffer as much.

https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html

and also

https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
( read the documentation for the parameters pti=, spectre_v2=, spec_store_bypass_disable= )

You can disable kernel microcode updating (which will revert to the microcode in your BIOS/UEFI) using the dis_ucode_ldr parameter on the *bootloader* (grub, etc). Instructions are in intel-microcode's README file in /usr/share/doc/intel-microcode

You can just install an older version of the intel-microcode package and reboot, if you want to switch to an earlier microcode update **provided that your system firmware (BIOS/UEFI) does not have a newer microcode**.

[Henrique de Moraes Holschuh (hmh)]

For guidance on how to set kernel parameters:
https://wiki.ubuntu.com/Kernel/KernelBootParameters

[Henrique de Moraes Holschuh (hmh)]

Setting to incomplete, since the fact that spectre, ssb and l1tf mitigations are expensive (and utterly so for some workloads) is well documented and well known, and there is nothing we can suggest unless we know the under-the-hood details of the affected workload.

[Launchpad Janitor (janitor)]

[Expired for intel-microcode (Ubuntu) because there has been no activity for 60 days.]