Enable late loading of microcode by default

[Dimitri John Ledkov]

[Impact]

 * An explanation of the effects of the bug on users and

 * justification for backporting the fix to the stable release.

 * In addition, it is helpful, but not required, to include an
   explanation of how the upload fixes this bug.

 * Normally intel microcode is applied "early" for an uncompressed prepended initramfs archive. However, on systems booting without an initrd, or a missbuilt one, microcode might not get applied. In that case, we need to attempt loading microcode late which may give users security protection against CPU vulnerabilities which they might otherwise be lacking. In an ideal world, everyone would apply their bios/OEM updates with microcode updates in a timely fashion and then we wouldn't need to update CPU microcode from userspace at all.

[Test Case]

 * Install updated package
 * Reobot
 * Observe early application of microcode

$ journalctl -b | grep microcode
Feb 12 12:02:48 ottawa kernel: microcode: microcode updated early to revision 0xd6, date = 2019-10-03

 * Remove /usr/share/initramfs-tools/hooks/intel_microcode to prevent correct generation of early microcode updates
 * Rebuild initrd with update-initramfs -u
 * Reboot
 * Observe in dmesg that late loading of microcode is performed

$ journalctl -b | grep microcode
Feb 12 12:32:54 ottawa kernel: TAA: Vulnerable: Clear CPU buffers attempted, no microcode
Feb 12 12:32:54 ottawa kernel: MDS: Vulnerable: Clear CPU buffers attempted, no microcode
Feb 12 12:32:54 ottawa kernel: microcode: sig=0x506e3, pf=0x20, revision=0xc6
Feb 12 12:32:54 ottawa kernel: microcode: Microcode Update Driver: v2.2.
Feb 12 12:32:57 ottawa kernel: microcode: updated to revision 0xd6, date = 2019-10-03
Feb 12 12:32:57 ottawa kernel: x86/CPU: CPU features have changed after loading microcode, but might not take effect.
Feb 12 12:32:57 ottawa kernel: microcode: Reload completed, microcode revision: 0xd6

(Note the lack of "early" in above messages)

[Regression Potential]

 * Application of microcode is a risky operation, especially if the cores are busy. Hence we prefer bios updates & early microcode updates, and those will remain the place. The late loading of microcode is really here for the cases were the previous two update strategies have failed. For example, from time to time, certain microcode updates are pulled or get blacklisted from late loading.

[Other Info]

 * The majority of our users on bare-metal machines boot correctly with early microcode updates.

[Dimitri John Ledkov (xnox)]

intel-microcode (3.20191115.1ubuntu3) focal; urgency=medium

  * Ship tmpfiles.d snippet to attempt late loading of microcode during
    boot, in case early loading of microcode did not happen. Early
    microcode loading might not happen if booting without initramfs or a
    missbuilt one.

 -- Dimitri John Ledkov <email address hidden> Wed, 12 Feb 2020 12:37:30 +0000

[Timo Aaltonen (tjaalton)]

Hello Dimitri, or anyone else affected,

Accepted intel-microcode into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/intel-microcode/3.20191115.1ubuntu0.19.10.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Beattie (sbeattie)]

Hey Dimitri,

Can you expand on what problems/situations where you're actually seeing late loading be a solution?

The reason I ask is that every communication I've had with Intel has indicated that late loading is risky and should not be used. The reason for this is that performing late loading on a running system can result in race conditions where cpu cores have different values for MSRs/cpu flags, or even have them disappear momentarily while the microcode is loading. This can cause a variety of problems for virtual machine hosts/hypervisors.

Also this statement:

  "For example, from time to time, certain microcode updates are pulled or get blacklisted from late loading."

isn't really a reason to do late loading.

Finally, why is this being done via tmpfiles.d(5)? If we're really going to do this, should it not be its own systemd unit, rather than hijacking something that isn't related?

Thanks.

[Dimitri John Ledkov (xnox)]

I'm trying to fix the case when users/manufacturers failed to provide/install uefi capsule update on the motherboard, failed to first-boot with up to date microcode, and thus remain unsecured, whilst one can install microcode. In bare-metal cloud context, late loading of microcode can be done as the line of last defence to apply microcode.

[Henrique de Moraes Holschuh (hmh)]

On Fri, 14 Feb 2020, Steve Beattie wrote:
> The reason I ask is that every communication I've had with Intel has
> indicated that late loading is risky and should not be used. The reason

Well, it is risky, it is actively discouraged, and regular users are
NEVER expected to come anywhere close to late loading.

The target audience of late loading is, as far as I know, gigantic cloud
provider senior engineers with proper NDAs signed with Intel and access
to relevant support channels and non-public information.

I am not adding this change to Debian, FWIW.

--
  Henrique Holschuh

[Steve Beattie (sbeattie)]

The version in eoan was superceded by the 20200609 release. In focal and groovy, this change was reverted in 3.20200609.0ubuntu0.20.04.2 because the tmpfiles.d approach, in addition to attmepting to late load early in the boot process, also caused late loading to trigger during package installation.

Enabling late loading by default also makes it more difficult to recover from a problematic microcode update, attempting to boot an earlier kernel/initramfs with a known-good microcode to recover will be thwarted by the late loaded problematic microcode.