intel-microcode is out of date, version 20170707 fixes errata on 6th and 7th generation platforms

Bug #1700373 reported by C de-Avillez on 2017-06-25
360
This bug affects 56 people
Affects Status Importance Assigned to Milestone
intel-microcode (Ubuntu)
Status tracked in Artful
Trusty
Undecided
Unassigned
Xenial
Undecided
Dave Chiluk
Yakkety
Undecided
Unassigned
Zesty
Undecided
Dave Chiluk
Artful
Undecided
Unassigned

Bug Description

[Impact]

* A security fix has been made available as part of intel-microcode
* It is advisable to apply it
* Thus an SRU of the latest intel-microcode is desirable for all stable releases

[Test Case]

* Upgrade intel-microcode package, if it is already installed / one is running on Intel CPUs

* Reboot and verify no averse results, and/or that microcode for your cpu was loaded as expected.

* Ocaml crash reproducer

Download report.tar.gz from https://caml.inria.fr/mantis/view.php?id=7452 and place in your schroot scratch directory.
$ mk-sbuild artful --arch=amd64
$ schroot -c artful -u root
// Artful was chosen as it contains the required versions of Ocaml for the reproducer.
$ apt install ocaml opam ocaml-findlib m4
$ opam init
$ opam install extprot
$ eval `opam config env`
$ while ocamlfind opt -c -g -bin-annot -ccopt -g -ccopt -O2 -ccopt -Wextra -ccopt '-Wstrict-overflow=5' -thread -w +a-4-40..42-44-45-48-58 -w -27-32 -package extprot test.ml -o test.cmx; do echo "ok"; done

[Test case reporting]
* Please paste the output of:

dpkg-query -W intel-microcode
grep -E 'model|stepping' /proc/cpuinfo | sort -u
journalctl -k | grep microcode

[Regression Potential]
Microcode are proprietary blobs, and can cause any number of new errors and regressions. Microcode bugs have been reported before, therefore longer than usual phasing and monitoring of intel-microcode bugs should be done with extra care.

Additional notes from ~racb, wearing an ~ubuntu-sru hat:

SRU verification needs to take care to consider CPUs actually tested. We should have a representative sample of CPUs tested in SRU verification reports before considering release to the updates pockets.

Given the potential severity of regressions, we should keep this in the proposed pockets for longer than the usual minimum ageing period. Let's have users opt-in to this update first, and only recommend it once we confidence that a reasonable number (and representative CPU sample) of opted-in users have not hit any problems.

Testers: please mark verification-done-* only after you consider that the above additional requirements have been met.

[Other]
caml discussion describing test case to reproduce the crash.
https://caml.inria.fr/mantis/view.php?id=7452

* I did not backport the full debian/changelog, as some of the changes were ommitted for SRU purposes, and I don't like the idea of modifying the changelog of others.

* I did not backport this below change but I feel as though the SRU team should evaluate including it. I left it out due to the change as little as possible guidance from the SRU team. Additionally we have already been shipping the microcode version that included this change for a long time. More information here
https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00030&languageid=en-fr

'''
# 0x206c2: Intel Westmere B1 (Xeon 3600, 5600, Core i7 2nd gen).
#
# When Intel released a fix for Intel SA-00030, they issued a MCU that
# bumps the minimum acceptable version of the Intel TXT ACMs in the
# TPM persistent storage. This permanently blacklists the vulnerable
# ACMs *even on older microcode* in order to make it somewhat harder
# to work around the security fix through a BIOS downgrade attack.
#
# It is possible that such a microcode update, when peformed by the
# operating system, could sucessfully trigger the TPM persistent
# storage update Intel intended to happen during firmware boot: we
# simply don't know enough to rule it out. Should that happen, Intel
# TXT will be permanently disabled. This could easily interact very
# badly with the firmware, rendering the system unbootable. If *that*
# happens, it would likely require either a TPM module replacement
# (rendering sealed data useless) or a direct flash of a new BIOS with
# updated ACMs, to repair.
#
# Blacklist updates for signature 0x206c2 as a safety net.
IUC_EXCLUDE += -s !0x206c2
'''

* I versioned the packages 3.20170511.1~ubuntu<release> as I feel this more appropriately reflects the contents of each package rather than simply incrementing the ubuntu version number.

=========================================================================

[Original bug report]

NB: I am *not* directly affected by this bug.

Henrique emailed a warning to Debian devel today [1] on a potentially serious issue with (sky|kaby)lake processors. Excerpt:

"This warning advisory is relevant for users of systems with the Intel
processors code-named "Skylake" and "Kaby Lake". These are: the 6th and
7th generation Intel Core processors (desktop, embedded, mobile and
HEDT), their related server processors (such as Xeon v5 and Xeon v6), as
well as select Intel Pentium processor models.

TL;DR: unfixed Skylake and Kaby Lake processors could, in some
situations, dangerously misbehave when hyper-threading is enabled.
Disable hyper-threading immediately in BIOS/UEFI to work around the
problem. Read this advisory for instructions about an Intel-provided
fix."

It is probably a good idea to:
(1) issue a warning to our users about this;
(2) update intel-microcode on all our supported releases

I leave the discussion on whether this can have security implications to others.

[1] https://lists.debian.org/debian-devel/2017/06/msg00308.html

ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: intel-microcode 3.20161104.1
ProcVersionSignature: Ubuntu 4.10.0-24.28-generic 4.10.15
Uname: Linux 4.10.0-24-generic x86_64
ApportVersion: 2.20.4-0ubuntu4.1
Architecture: amd64
CurrentDesktop: Unity:Unity7
Date: Sun Jun 25 10:14:19 2017
InstallationDate: Installed on 2017-05-26 (30 days ago)
InstallationMedia: Ubuntu 17.04 "Zesty Zapus" - Release amd64 (20170412)
SourcePackage: intel-microcode
UpgradeStatus: No upgrade log present (probably fresh install)

C de-Avillez (hggdh2) wrote :
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in intel-microcode (Ubuntu):
status: New → Confirmed
Edwin Khoo (edwinksl) wrote :

Relevant Hacker News comment about this issue: https://news.ycombinator.com/item?id=14630459

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in intel-microcode (Ubuntu Xenial):
status: New → Confirmed
Changed in intel-microcode (Ubuntu Yakkety):
status: New → Confirmed
Changed in intel-microcode (Ubuntu Zesty):
status: New → Confirmed
Changed in intel-microcode (Ubuntu Artful):
status: Confirmed → Fix Released
no longer affects: intel
description: updated
Dimitri John Ledkov (xnox) wrote :

@Nico

Please hide your comment, this bug will be used for Stable Release Update verification, and we would need users to test the packages from the -proposed pockets and not from your ppa.

Regards,

Dimitri.

Changed in intel-microcode (Ubuntu Zesty):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in intel-microcode (Ubuntu Yakkety):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in intel-microcode (Ubuntu Xenial):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in intel-microcode (Ubuntu Zesty):
status: Confirmed → In Progress
Changed in intel-microcode (Ubuntu Yakkety):
status: Confirmed → In Progress
Changed in intel-microcode (Ubuntu Xenial):
status: Confirmed → In Progress
Adrien Beau (adrienbeau) wrote :

I accidentally wrote a duplicate report of this bug: I had checked the bug list before reporting, but this bug was not visible because it was already marked as fixed (in Arty).

Since this bug will be used for Stable Release Updates, is there a way to still list it (with the default Launchpad filters) as long as a previous release remains In Progress?

Robie Basak (racb) wrote :

Some discussion on the approach to take in backporting here:

https://irclogs.ubuntu.com/2017/06/26/%23ubuntu-release.html#t15:46

It seems to me that there are two approaches:

A) Backport only the binary blobs, leaving everything else the same. This was the approach taken in the previous intel-microcode updates in 14.04, and what I was expecting when I began my review.

B) Backport the latest intel-microcode package wholesale. This is what xnox has uploaded for review.

I currently favour A, for reasons debated in the IRC discussion linked above. In short, I think that given that the packaging primarily carries the delivery mechanism, and the delivery mechanism necessarily interacts with other components such as the kernel, unnecessarily updating the delivery mechanism means unnecessarily introducing additional regression risk. I don't think I've been given any counter-example where approach A introduces a regression risk that approach B does not.

I'm declining to accept these uploads into -proposed for now, since I'm not convinced that this approach carries minimal regression risk.

I do agree that given the nature of the update we should probably go with an extra-long aging period in -proposed and slow phasing if possible regardless of approach. It would be nice to give users an opt-in to this update soon, however.

Nothing in my opinion rules out a longer term view on intel-microcode with more frequent updates to keep up with upstream on an ongoing basis to all stable releases. We could choose to do that now, with this update being the first one of those. If so, we should work out a fully documented procedure and exception (under the hardware enablement/environmental rationale). But we don't have that currently. Before accepting this set of backports I'd like to have that agreed so that we don't end up maintaining some exception that we don't want to keep.

I would be happy to follow approach A for now, and work out a full process for ongoing updates for next time. Or we can continue discussion and agree on a different approach.

Opinions welcome. Am I the only one on the SRU team with this view?

Rally (rallyspam) wrote :

We have Skylake and Kaby Lake 14.04 systems /w hyper threading. Can we add trusty to the backport list?

Robie Basak (racb) wrote :

16:58 <rbasak> xnox: not doing Trusty?

17:00 <xnox> rbasak, nope.

17:01 <xnox> rbasak, these new microcodes require early initramfs loading code; which is not in trusty. And I don't think intel-microcode is widely installed in trusty.

17:02 <xnox> rbasak, in xenial we have started automated intel-microcode installation with ubuntu-drivers

I'm not sure what "require early initramfs loading code" means exactly. Does that mean that approach B cannot work? What about approach A?

I think we should be clear to users on expectations for Trusty though. So I'll open a task, which we can set to Triaged, Won't Fix or In Progress as appropriate.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in intel-microcode (Ubuntu Trusty):
status: New → Confirmed
Rally (rallyspam) wrote :

My use case of 14.04 isn't home use but in an enterprise environment, I'm not easily able to get our systems to 16.04 LTS. What is the criteria used to determine when a fix gets backported to a LTS release? One of the reasons we went with 14.04 is because it's a LTS release with a 5 year lifecycle.

Jeremy Bicha (jbicha) wrote :

Rally, the policy is at https://wiki.ubuntu.com/StableReleaseUpdates

Ubuntu 14.04 LTS gets 5 years of security updates, but not every bugfix will be backported to that release.

My understanding of the somewhat high profile issue reported this weekend that triggered this bug and renewed interest in getting these microcode updates to Ubuntu 16.04 LTS is that the headline issue is only known to affect oCaml code. It's likely that most Ubuntu users do not have any installed oCaml code.

That doesn't mean the update is useless for most Ubuntu users. There are other processor fixes in the updates that may fix or workaround issues that do affect more users. (Intel does not really provide many details of what is changed in these updates.)

Marat Khalili (mkh-t) wrote :

*It is* a security update, unless proven otherwise. "Unpredictable System Behavior" is often just a polite name for hardware vulnerability.

I also run Trusty in enterprise environment. Particularly, I run it on a mission critical secondary network server (DNS, NTP etc.) intentionally paired to a primary Xenial machine so as not to create too homogeneous environment. Therefore I need it to be fixed in Trusty. Disabling HT is not very good workaround for i3-6100U in there too.

Robie Basak (racb) wrote :

09:04 <rbasak> SRU team: second opinion on bug 1700373 please?

09:14 <apw> rbasak, oh any particular aspect?

09:14 <rbasak> apw: my comment 10

09:15 <rbasak> apw: on the full package backport vs blob only question.

09:19 <apw> rbasak, i would feel the same, that backporting the content ony would be my expectation, as delivery is local to the series

09:19 <apw> rbasak, pretty much i agree with your comment en-toto

Rally (rallyspam) wrote :

Is the primary point of contention for a trusty backport the early initramfs microcode loading referenced in comment #12? Can we continue using the same mechanism previous trusty microcode updates used?

Also, it sounds like 14.04 trusty has the early microcode loading feature?

https://bugs.launchpad.net/intel/+bug/1084373/comments/11

Please disregard if the above link is something else but it would be good to know the technical details on what's preventing a trusty backport.

Amr Ibrahim (amribrahim1987) wrote :

A side note to the proposed SRU exception policy for CPU microcode:
Please also take into account the package amd64-microcode. It's the processor microcode firmware for AMD CPUs. It also fixes critical errata and provide stability and security updates.

Dave Chiluk (chiluk) wrote :

For those of you that really want Trusty fixed, please be aware, that the preferred way to resolve this kind of issue is to upgrade the EFI Firmware on your machine.

Also I just checked the source code for initramfs-tools, Yingying Zhao (yingying-zhao) was wrong in
https://bugs.launchpad.net/intel/+bug/1084373/comments/11
as the trusty version definitively does not have the early microcode patch.
https://anonscm.debian.org/git/kernel/initramfs-tools.git/commit/?id=d5f4cd6
Additionally the version where that is committed has changed significantly from Trusty such that backporting that doesn't look advisable either.

Xnox has made a correct assessment by not wanting to push this into Trusty. Please stop voting for trusty. If you insist on running Trusty please directly upgrade your EFI.

Thanks,

Adrien Beau (adrienbeau) wrote :

Dave, not all vendors ship updated microcode in a timely manner. This is rarely the case on consumer motherboards, hopefully the situation is better on servers.

Also, there is a debate here, between the maintainer of the package and the SRU team. If I understand correctly, the package includes both part of the delivery mechanism (which also depends on the kernel version) and the Intel binary blobs. The maintainer wants to keep it that way (less of a burden going forward), but this means it is very hard to backport it to Trusty. The SRU team is interested in backporting only the Intel binary blobs, which would also work for Trusty, but this would fork the way the package is created, and means more work going forward.

There is also the matter of doing this properly, not just fixing the issue currently under the spotlight, but also ensuring more timely microcode updates in the long run.

As far as I remember, the switch to a new firmware uploading mechanism was due to the fact that newer firmware disabled TSX instructions completely on Haswell, thus causing segfaults of applications linked against libpthread. I.e. patching firmware later than "very early on boot" was too late at least for Haswell.

This consideration does not apply directly to Trusty, as Trusty's glibc never used hardware lock elision. However, this might be the case for containers and chroots (including non-Ubuntu ones) running on Trusty. As far as I understand, Ubuntu glibc was later rebuilt to never use hardware lock elision, because it exposed too many double-unlocking bugs in third-party software.

My conclusion is that it is 100% unsafe to update firmware blobs in Trusty if there is a possibility that any non-Ubuntu container gets started before the firmware is uploaded to the CPU. If such situation is impossible, then please treat this comment as a no-op.

Uqbar (uqbar) wrote :

If you understood correctly (please add references) and in my opinion, mixing INTEL/AMD CPU microcode with third party stuff (either software or extra documentation) is not less than a thinko, when not a real bug.

Microcode is updated by manufacturers.
Software and extra documentation is done by package maintainers.

Those need to be separated, as we need to be able to update the microcode (especially in case of security updates), no matter whether I am running 14.04, 16.04 or whatever else release.

Unless someone thinks that a complete system upgrade is the solution: upgrade (and possible break) a complete system in order to update this very single package.

Please, remind this discussion is related to this very type of packages, not to all of them in the repos!

At the moment I am applying this workaround:

1. Search and download the latest deb package from Debian.org at https://packages.debian.org/search?keywords=intel-microcode&searchon=names&suite=all&section=all

2. Download it

3. Install it with "sudo dpkg -i <package full name.deb>".

Uqbar (uqbar) wrote :

In my servers fimrware is uploaded to CPUs early in the boot:

[ 0.000000] microcode: microcode updated early to revision 0x22, date = 2017-01-27

This is the very first line in the kernel log I can read.
I cannot see any possibility that "any non-Ubuntu container" (actually anything else) is started earlier than this. Isn't it?

Changed in intel-microcode (Ubuntu Zesty):
assignee: Dimitri John Ledkov (xnox) → nobody
Changed in intel-microcode (Ubuntu Xenial):
assignee: Dimitri John Ledkov (xnox) → nobody
Changed in intel-microcode (Ubuntu Yakkety):
assignee: Dimitri John Ledkov (xnox) → nobody
status: In Progress → Confirmed
Changed in intel-microcode (Ubuntu Xenial):
status: In Progress → Confirmed
Changed in intel-microcode (Ubuntu Zesty):
status: In Progress → Confirmed

Uqbar: if your servers are running Trusty, then thanks for the confirmation that too-late update is not an issue.

Adrien Beau (adrienbeau) wrote :

Uqbar, what version of Ubuntu are you using, and what kernel version?

> I cannot see any possibility that "any non-Ubuntu container" (actually anything else) is started earlier than this. Isn't it?

Quite simply, the machine could be running any kind of supervisor (Xen, Hyper-V, ESX...), which could change the microcode of the CPU, and then much later boot Ubuntu in a VM. Most likely, Ubuntu detects such cases and there is no problem, but I believe this is what Alexander E. Patrakov had in mind when mentioning non-Ubuntu containers.

Robie Basak (racb) wrote :

There has been no further feedback from other members of the SRU team (neither in this bug nor in the ubuntu-release@ ML thread). In order to make progress I'm proceeding with my decision as the position of the SRU team. We can re-evaluate if new technical considerations come to light, or if others in the SRU team or TB want to interject.

On this basis we are not proceeding with xnox's backports for Xenial and Yakkety, since they change more than just the microcode.

xnox's Zesty upload I reluctantly accept. It doesn't introduce any packaging changes (as it's close enough to the latest in Artful and in Debian sid) so I concede that it cannot introduce any additional regression risk over just a blob update. It is stylistically a backport though (eg. changelog, version string), rather than a blob update, so will end up looking different from Xenial and Yakkety if X and Y do get blob-only updates in the end. I don't like this, but as xnox has pointed out pushing the update to Zesty will help with phasing the update to mitigate risk to users, so I am accepting it.

Please note that packages in Ubuntu are team maintained. As xnox has unassigned himself for now, I'll seek another developer to prepare the updates for Xenial and Yakkety. I would do it myself, but I'm not supposed to wear both hats.

Once we have another developer looking at this, we can examine the Trusty situation and make a decision about Trusty.

description: updated

Hello C, or anyone else affected,

Accepted intel-microcode into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/intel-microcode/3.20170511.1~ubuntu17.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty.If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in intel-microcode (Ubuntu Zesty):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-zesty

Maybe mine is just wishful thinking, but microcode update is hardware initialization, containers boot is software. The former should always happen before the latter.

If you have all this computing infrastructure (hypervisors+guests+containers) then it'd be easy to replicate a container on another (virtual) machine and test the new microcode.
This should be done for any update, not just microcode. An apache update could break your web application. Couldn't it?

FWIW, since I am a completely neutral party (Debian maintainer), here are some points that may help the Ubuntu teams make a decision:

1. Hypervisors must not (and don't) allow a guest to supply microcode updates, they're blocked at the WRMSR handler. This is required behavior, because it is trivial for a guest to attack the whole host system using microcode updates, resulting in denial-of-service (core hangs) and specially crafted microcode containers (whole box hangs, maybe crashes/reboots, microcode downgrade as a step in a complex attack).

Container-based systems cannot filter "guests" (containers) at the WRMSR opcode level like hypervisors can, and therefore must somehow block all entry points to the kernel that could trigger a microcode update from untrusted data. Beware the MSR.ko module.

2. In Debian, we are not aware of any issues with the Trusty's method of microcode updates (late updates) other than the ones related to Intel TSX.

However, late microcode updates have increased risk of regressions since they happen after the kernel and modules have initialized MSRs, etc. We have seen such issues related to *BIOS*-initialized MSRs on Xeon E5v3 (which required further microcode updates to fix), but we have not observed anything related to kernel-initialized MSRs. So, this increased risk is a theoretical issue at the moment, AFAIK.

3. Almost nobody tests late updates anymore. All large distros switched to early updates a couple years ago, AFAIK.

4. What Debian did for Debian wheezy (which uses the same outdated method as Ubuntu Trusty to update microcode) was to have two sets of microcode-updating packages and kernel packages (in this case, using wheezy-backports), and drop support for Haswell and later processors using the old method.

However, this whole thing did not work very well and now the Wheezy-LTS team will have to very much make the same kind of decision you must make for Ubuntu Trusty.

Marat Khalili (mkh-t) wrote :

Concerning microcode update time in Trusty, real test (with linux-generic-lts-xenial installed):
---
marat@CM01:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty

marat@CM01:~$ uname -a
Linux CM01.administration.intranet.rqc.ru 4.4.0-81-generic #104~14.04.1-Ubuntu SMP Wed Jun 14 12:45:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

marat@CM01:~$ dmesg | nl | grep -i microcode
     1 [ 0.000000] microcode: CPU0 microcode updated early to revision 0x29, date = 2013-06-12
   226 [ 0.080489] microcode: CPU1 microcode updated early to revision 0x29, date = 2013-06-12
   229 [ 0.083195] microcode: CPU2 microcode updated early to revision 0x29, date = 2013-06-12
   230 [ 0.085331] #3<6>[ 0.085831] microcode: CPU3 microcode updated early to revision 0x29, date = 2013-06-12
   670 [ 1.262792] microcode: CPU0 sig=0x206a7, pf=0x2, revision=0x29
   671 [ 1.262799] microcode: CPU1 sig=0x206a7, pf=0x2, revision=0x29
   672 [ 1.262836] microcode: CPU2 sig=0x206a7, pf=0x2, revision=0x29
   673 [ 1.262877] microcode: CPU3 sig=0x206a7, pf=0x2, revision=0x29
   674 [ 1.262960] microcode: Microcode Update Driver: v2.01 <email address hidden>, Peter Oruba
   812 [ 2.696416] [drm] Loading CAICOS Microcode
---
So it's in the beginning, though not all quite 0.000000. (Last line is apparently unrelated to CPU.)

As for microcode updates _from_ container, I thought it should go without saying that it must be prohibited.

Robie Basak (racb) wrote :

Thank you Henrique, that is very helpful.

Dave Chiluk (chiluk) wrote :

Tested http://launchpadlibrarian.net/325879940/intel-microcode_3.20170511.1~ubuntu17.04.0_amd64.deb
into Xenial+rolling-hwe. So far seems ok.

[ 0.000000] microcode: microcode updated early to revision 0xba, date = 2017-04-09
[ 0.000000] Linux version 4.8.0-56-generic (buildd@lcy01-33) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #61~16.04.1
-Ubuntu SMP Wed Jun 14 11:58:22 UTC 2017 (Ubuntu 4.8.0-56.61~16.04.1-generic 4.8.17)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.8.0-56-generic.efi.signed root=UUID=5c51331a-6c14-4bd6-86b6-00d4b00f4da6 ro quiet
 splash vt.handoff=7
...
[ 0.876613] microcode: sig=0x506e3, pf=0x2, revision=0xba
[ 0.876934] microcode: Microcode Update Driver: v2.01 <email address hidden>, Peter Oruba

Eugene San (eugenesan) on 2017-06-28
tags: added: lts
summary: - Please update microcode to version 20170511 on all supported platforms
+ intel-microcode should be updated, version 20170511 fixes severe errata
+ on 6th and 7th generation platforms
summary: - intel-microcode should be updated, version 20170511 fixes severe errata
- on 6th and 7th generation platforms
+ intel-microcode should be updated for LTS releases, version 20170511
+ fixes severe errata on 6th and 7th generation platforms

@Eugene San

We're updating non-LTS releases too. For example a Zesty SRU is in progress.

summary: - intel-microcode should be updated for LTS releases, version 20170511
- fixes severe errata on 6th and 7th generation platforms
+ intel-microcode is out of date, version 20170511 fixes severe errata on
+ 6th and 7th generation platforms

Also "severe" is subjective. So far we only have reports of this affecting ocaml users, which isn't the majority of Ubuntu users. I understand that many are concerned because of problems this bug *may* cause, but let's keep some perspective on this.

summary: - intel-microcode is out of date, version 20170511 fixes severe errata on
- 6th and 7th generation platforms
+ intel-microcode is out of date, version 20170511 fixes errata on 6th and
+ 7th generation platforms
Dave Chiluk (chiluk) on 2017-06-29
Changed in intel-microcode (Ubuntu Yakkety):
assignee: nobody → Dave Chiluk (chiluk)
Changed in intel-microcode (Ubuntu Xenial):
assignee: nobody → Dave Chiluk (chiluk)

On Ubuntu 17.04 zesty the proposed microcode package installs and loads fine:
$ dpkg-query -W intel-microcode
intel-microcode 3.20170511.1~ubuntu17.04.0
$ grep -E 'model|stepping' /proc/cpuinfo | sort -u
model : 78
model name : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping : 3
$ journalctl -k|grep microcode
Jul 01 14:21:47 thunder kernel: microcode: microcode updated early to revision 0xba, date = 2017-04-09
Jul 01 14:21:47 thunder kernel: microcode: sig=0x406e3, pf=0x80, revision=0xba
Jul 01 14:21:47 thunder kernel: microcode: Microcode Update Driver: v2.2.

I have confirmed that it also fixes the original testcase from the OCaml bugtracker (with the old microcode it was crashing quite soon).
$ eval $(opam config env)
The OCaml toplevel, version 4.04.1
$ while ocamlfind opt -c -g -bin-annot -ccopt -g -ccopt -O2 -ccopt -Wextra -ccopt '-Wstrict-overflow=5' -thread -w +a-4-40..42-44-45-48-58 -w -27-32 -package extprot test.ml -o test.cmx; do echo "ok"; done
ok
ok
ok
ok
ok
...

Running the perl script posted on the Debian mailing list shows all OK too:
cpu 0: Your CPU is affected, but your microcode is new enough
cpu 1: Your CPU is affected, but your microcode is new enough
cpu 2: Your CPU is affected, but your microcode is new enough
cpu 3: Your CPU is affected, but your microcode is new enough

Note: I've been using the 20170511 version of microcode using the package from artful for about 6 days, and haven't noticed any problems.

tags: added: verification-done-zesty
removed: verification-needed-zesty
dorpm (dorpmueller) wrote :

On Ubuntu 16.04 till yet no update. When will this be available?

Dave Chiluk (chiluk) wrote :

I just pushed uploads for this into the x, and y queues. I wanted to hold off for a few days to give those on zesty/artful to sniff test this before I pushed on the SRU.

I have tested this on X+skylake and X+sandy-bridge-e *(that was just to make sure that we didn't break non skylake and kabylake cpus).

#wosru

Dave Chiluk (chiluk) wrote :

I should also mention that we will be following Debian's lead and not updating this for Trusty. Lack of early upload being one of the big reasons for this.

description: updated
Changed in intel-microcode (Ubuntu Trusty):
status: Confirmed → Won't Fix
Robie Basak (racb) wrote :

> I should also mention that we will be following Debian's lead and not updating this for Trusty.

Who is "we", please? Is this a decision of the Ubuntu Foundations team? Or are you saying that you won't work on it, but it's open for others to attempt?

Robie Basak (racb) wrote :

FTR, after discussion with Dave on IRC, I rejected his other uploads from the queue because they were straight backports. The SRU team has already rejected this due to a lack of mitigation for regression risk to users, but Dave had missed the thread in ubuntu-release@ about this.

The current status of this bug is that we're still waiting for someone (or some team) to contribute acceptable updates for Xenial and Yakkety. As far as I am aware, Trusty is still undecided, but seems unlikely due to the early userspace issue.

Marat Khalili (mkh-t) wrote :

Pardon my ignorance, can someone please clarify in more details meaning of "require early initramfs loading code" from comment #12? Because clearly some people will go ahead and update it in Trusty anyways.

Judging from other comments new microcode may break something that is unlikely to be present in Trusty at all, and especially unlikely to be running in the first hundred milliseconds when microcode is actually updated. Does not sound very convincing.

Robie Basak (racb) wrote :

@Marat

10:58 <rbasak> xnox: thanks, that's very useful to know. What's the the difference between early initramfs and normal runtime from the kernel/hardware perspective? I didn't think there was any distinction? But there has been talk of Trusty not having early initramfs support. I don't follow why that's a thing.
10:59 <xnox> rbasak, early micocode loading means that microcode is appended to the initramfs as an extra cpio archive. The kernel looks at the initramfs, notices there is microcode appended, and then it loads the microcode before executing init of the initramfs.
10:59 <xnox> rbasak, meaning that microcode is loaded "early", before any userspace process is started.
11:00 <xnox> rbasak, this is relevant in the context of e.g. lock ellision where microcode update _removes CPU instructions_
11:00 <rbasak> Ah. That makes sense - thanks!
11:00 <xnox> and e.g. loaded shared libraries already did the checks if they can use something, continue to try to use them, and segfault.

> Does not sound very convincing.

For an SRU, the burden of "convincing" is reversed. Convince us that this issue *won't* regress users before we recommend an update to them, not the other way round.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package intel-microcode - 3.20170511.1~ubuntu17.04.0

---------------
intel-microcode (3.20170511.1~ubuntu17.04.0) zesty; urgency=medium

  * Backport of new upstream microde release to address Hyper Threading
    bug. LP: #1700373.

intel-microcode (3.20170511.1) unstable; urgency=medium

  * New upstream microcode datafile 20170511
    + Updated Microcodes:
      sig 0x000306c3, pf_mask 0x32, 2017-01-27, rev 0x0022, size 22528
      sig 0x000306d4, pf_mask 0xc0, 2017-01-27, rev 0x0025, size 17408
      sig 0x000306f2, pf_mask 0x6f, 2017-01-30, rev 0x003a, size 32768
      sig 0x000306f4, pf_mask 0x80, 2017-01-30, rev 0x000f, size 16384
      sig 0x00040651, pf_mask 0x72, 2017-01-27, rev 0x0020, size 20480
      sig 0x00040661, pf_mask 0x32, 2017-01-27, rev 0x0017, size 24576
      sig 0x00040671, pf_mask 0x22, 2017-01-27, rev 0x0017, size 11264
      sig 0x000406e3, pf_mask 0xc0, 2017-04-09, rev 0x00ba, size 98304
      sig 0x000406f1, pf_mask 0xef, 2017-03-01, rev 0xb000021, size 26624
      sig 0x000506e3, pf_mask 0x36, 2017-04-09, rev 0x00ba, size 98304
    + This release fixes undisclosed errata on the desktop, mobile and
      server processor models from the Haswell, Broadwell, and Skylake
      families, including even the high-end multi-socket server Xeons
    + Likely fix the TSC-Deadline LAPIC errata (BDF89, SKL142 and
      similar) on several processor families
    + Fix erratum BDF90 on Xeon E7v4, E5v4(?) (closes: #862606)
    + Likely fix serious or critical Skylake errata: SKL138/144,
      SKL137/145, SLK149
    * Likely fix nightmare-level Skylake erratum SKL150. Fortunately,
      either this erratum is very-low-hitting, or gcc/clang/icc/msvc
      won't usually issue the affected opcode pattern and it ends up
      being rare.
      SKL150 - Short loops using both the AH/BH/CH/DH registers and
      the corresponding wide register *may* result in unpredictable
      system behavior. Requires both logical processors of the same
      core (i.e. sibling hyperthreads) to be active to trigger, as
      well as a "complex set of micro-architectural conditions"
  * source: remove unneeded intel-ucode/ directory
    Since release 20170511, upstream ships the microcodes both in .dat
    format, and as Linux-style split /lib/firmware/intel-ucode files.
    It is simpler to just use the .dat format file for now, so remove
    the intel-ucode/ directory. Note: before removal, it was verified
    that there were no discrepancies between the two microcode sets
    (.dat and intel-ucode/)
  * source: remove superseded upstream data file: 20161104

 -- Dimitri John Ledkov <email address hidden> Mon, 26 Jun 2017 16:22:13 +0100

Changed in intel-microcode (Ubuntu Zesty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for intel-microcode has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

I have repackaged this again based on the discussion from the ubuntu-release mailing list discussion. I have avoided any other changes that didn't appear absolutely necessary. I have uploaded the resulting packages to my ppa available here

https://launchpad.net/~chiluk/+archive/ubuntu/1700373

I would appreciate any additional testing I can get on these packages. Please keep your testing succinct to the following information.

I have tested
Skylake with signature 0x506e3 + 4.8.0-58-generic kernel + xenial = revision 0xba
Kabylake with signature 0x906e9 + 4.10.0.27 kernel + xenial = revision 0x5e
Ivybridge with signature 0x000306a9 + 4.4.0-83-generic + xenial = revision 0x1c
Ivybridge with signature 0x000306a9 + 4.8.0-58-generic + yakkety = revision 0x1c

I have also again uploaded the correctly versioned packages to the Upload queues for X and Y. I'll be updating the SRU template shortly as well.

description: updated

Relevant information:

I have uploaded to Debian yesterday, and it has already been auto-migrated to Ubuntu Artful Aardvark, the newest intel-microcode release (base date: 20170707).

This new upstream release fixes the hyper-threading issue (as well as several other undisclosed issues) on Skylake-X and Kaby Lake processors. It also has all the previous fixes from the previous releases (so it fixes the other Skylakes as well).

Dave Chiluk (chiluk) wrote :

@hmh, thanks for the heads up.

@rbasak, As the X and Y uploads haven't made it through the SRU yet. Do you want me to repackage with 20170707?

David Jordan (dmj726) wrote :

The 20170707 version would be ideal to SRU for Xenial (and Zesty) since it fixes the hyperthreading defect on Skylake, Skylake-X, and Kabylake. The Kabylake fix is especially important. There is a lot of demand from users for a fix, and bios updates really aren't an ideal way to deliver it.

Robie Basak (racb) wrote :

@Dave

Yes - please repackage. I see no reason to force 20170511 instead of 20170707. But I would prefer to release (after verification) Y and Z before releasing X.

In terms of tracking bugs, shall we retitle this bug for 20170707 and reopen it for Zesty? As there's nothing in flight at the moment, that would seem the easiest way to manage this to me. Any regression for Z if one is found would be tracked in a separate bug anyway, so I think this would be the least confusing.

Dave Chiluk (chiluk) wrote :

I have packaged up 20170707 for testing in my ppa available

https://launchpad.net/~chiluk/+archive/ubuntu/1700373

These are versioned as

Z=3.20170511.1~ubuntu17.04.0+lp1700373
Y=3.20160714.1+lp1700373b2
X=3.20151106.1+lp1700373b2

I'd appreciate some help testing these before I push on the SRU.

I've also uploaded correctly versioned packages for x, y, z to their appropriate queues.

Changed in intel-microcode (Ubuntu Zesty):
status: Fix Released → Confirmed
assignee: nobody → Dave Chiluk (chiluk)
tags: added: verification-done-artful
removed: verification-done-zesty verification-needed
Dave Chiluk (chiluk) wrote :

Testing done.
X + 4.10 + Sandybridge-E + sig=0x206d6, pf=0x4, revision=0x619 => 0x619
X + 4.4 + Ivy + sig=0x306a9, pf=0x10, revision=0x1c => 0x1c
Y + 4.8 + Ivy + sig=0x306a9, pf=0x10, revision=0x1c => 0x1c
Z + 4.10 + Ivy + sig=0x306a9, pf=0x10, revision=0x1c => 0x1c
Z + 4.10 + Skylake + sig=0x406e3 pf=0x80 revision=0xba => 0xba
X + 4.10 + Kabylake + sig=0x906e9 pf=0x20 revision=0x5e => 0x5e
A + 4.11 + Lynnfield sig=0x106e5, pf=0x20, revision=0x5 => 0x5

So it's a lot of the same compared to previous microcode levels on my mostly old hardware *(which was expected). Also my only Kabylake machine is at a sufficiently new bios revision that microcode 0x5e is already present. I'd appreciate anyone with skylake or kabylake that hasn't yet upgrade firmware to please test out the packages in my ppa. Thanks in advance.

@rbasak, yes according to the changelogs for 20170707, it contains the fixes for Kabylake and skylake-E/X.

summary: - intel-microcode is out of date, version 20170511 fixes errata on 6th and
+ intel-microcode is out of date, version 20170707 fixes errata on 6th and
7th generation platforms
description: updated
Marat Khalili (mkh-t) wrote :

Well, I tried intel-microcode_3.20151106.1+lp1700373b2_amd64.deb on Trusty with 4.4 and it worked. Of course, in someone's case it may start a thermonuclear war, so use this information with care.

Dmitrii Shcherbakov (dmitriis) wrote :

Tested on 17.04 with a 4.13-rc1 kernel + a pinned microcode package from artful:

➜ linux git:(5771a8c08880) ✗ apt policy intel-microcode
intel-microcode:
  Installed: 3.20170707.1
  Candidate: 3.20170707.1
  Version table:
 *** 3.20170707.1 500
        500 http://ru.archive.ubuntu.com/ubuntu artful/restricted amd64 Packages
        100 /var/lib/dpkg/status
     3.20170511.1~ubuntu17.04.0 990
        990 http://archive.ubuntu.com/ubuntu zesty-updates/restricted amd64 Packages
     3.20161104.1 990
        990 http://archive.ubuntu.com/ubuntu zesty/restricted amd64 Packages

➜ ~ uname -r
4.13.0-rc1

➜ ~ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 158
Model name: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
Stepping: 9
CPU MHz: 2800.000
CPU max MHz: 3800,0000
CPU min MHz: 800,0000
BogoMIPS: 5616.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp

➜ ~ dmesg | grep microcode
[ 0.000000] microcode: microcode updated early to revision 0x5e, date = 2017-04-06
[ 2.621961] microcode: sig=0x906e9, pf=0x20, revision=0x5e
[ 2.622083] microcode: Microcode Update Driver: v2.2.

My system definitely does not have 0x5e by default, the previous microcode version was 0x48:

[ 2.561730] microcode: sig=0x906e9, pf=0x20, revision=0x48
[ 2.561811] microcode: Microcode Update Driver: v2.2.

The microcode update also resulted in absence of a 'Firmware Bug' message due to TSC_DEADLINE APIC mode usage:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bd9240a18edfbfa72e957fc2ba831cf1f13ea073

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kernel/apic/apic.c?id=c6e9f42bbeecbc10cd4fbcca474b5859aba1de67#n386

Thanks!

Robie Basak (racb) wrote :

15:38 <rbasak> chiluk: around? Reviewing intel-microcode.

15:39 <rbasak> In principle the change seems OK, but I'm confused by things mismatching the descriptions in debian/changelog

15:39 <rbasak> "source: remove unneeded intel-ucode/ directory" - removed from where? I don't see this in the current packages for Xenial nor in Zesty.

15:40 <rbasak> "source: remove superseded upstream data file: 20170511" - this is not true for Xenial. You remove other files instead.

15:41 <rbasak> Though if we're removing superseded files, why do some remain? Are we following a pattern of leaving old files there, or removing old files? I'm not sure I follow why your proposed SRUs appear to do both. Is there some nuance I'm missing?

Dave Chiluk (chiluk) wrote :

The changelog entry is there to match the artful changelog entry *(it applies completely correctly for zesty).

As for the dat files. I guess my changelog entry would have been more explicit had I said sync instead of backport.

" * Backport of new upstream microcode release to address Hyper Threading
    bug. This is mostly a sync of the dat files from debian version
    3.20170707.1 (LP: #1700373) "

It should be noted that these dat files are never exposed via the bin package to the user.

As for the rest of the changelog entry, I thought about not including the debian changelog entry in the x upload, but I like to error on the side of more information, and this vvvvvv part of the changelog entry matches identically to zesty and artful.
" * New upstream microcode datafile 20170707
    + New Microcodes:
      sig 0x00050654, pf_mask 0x97, 2017-06-01, rev 0x2000022, size 25600
      sig 0x000806e9, pf_mask 0xc0, 2017-04-27, rev 0x0062, size 97280
      sig 0x000806ea, pf_mask 0xc0, 2017-05-23, rev 0x0066, size 95232
      sig 0x000906e9, pf_mask 0x2a, 2017-04-06, rev 0x005e, size 97280
    + This release fixes the nightmare-level errata SKZ7/SKW144/SKL150/
      SKX150 (Skylake) KBL095/KBW095 (Kaby Lake) for all affected Kaby
      Lake and Skylake processors: Skylake D0/R0 were fixed since the
      previous upstream release (20170511). This new release adds the
      fixes for Kaby Lake Y0/B0/H0 and Skylake H0 (Skylake-E/X).
    + Fix undisclosed errata in Skylake H0 (0x50654), Kaby Lake Y0
      (0x806ea), Kaby Lake H0 (0x806e9), Kaby Lake B0 (0x906e9)
  * source: remove unneeded intel-ucode/ directory
  * source: remove superseded upstream data file: 20170511
"

If you deem it crucially important, we could change that last line to
" * source: synced data files with debian version 20170707.1
"

Robie Basak (racb) wrote :

Hi Dave,

Thank you for going through this. I think it's OK if you want to keep the set of dat files available in sync with the set available in the development release. But then the changelog entry should make this clear.

I expected the changelog entry to match the diff that I was reviewing. When users view the entry, they would then expect the entry to match what they are about to receive (be it source or binaries).

For backports it's often easier to base the changelog on one from the version that's being backported and add one further entry explaining the backport. In this case my previous paragraph would still be accurate: the user would see all the changes landing in the stable release, and reviewers would also see a diff that corresponded to them.

However we have agreed that this update will be a cherry-pick, not a backport, so this wouldn't apply anyway.

> ...I thought about not including the debian changelog entry in the x upload, but I like to error on the side of more information, and this vvvvvv part of the changelog entry matches identically to zesty and artful.

I agree with the sentiment. More information is fine. However inaccurate or misleading information is not. So by all means use the information in the changelog entry from Artful as a starting point, but you do need to remove or correct parts that no longer apply.

Given that we aren't backporting the packaging from Artful, I don't think it makes sense to include lines like "source: remove unneeded intel-ucode/ directory". As far as I can find, this isn't happening at all to users, neither in Xenial nor Zesty, and isn't represented in the diff. I can't find this directory in any sources anywhere (just looking at Ubuntu).

> If you deem it crucially important...

It's hardly crucially important, but I do think it's reasonable to expect that the changelog is accurate against what is actually going on, I think it's reasonable for users to expect this too, and I believe it's the SRU team's job to maintain this standard.

Please correct as follows (or discuss further if you want to do something else):

1) Remove the comment about removing intel-ucode/, as I can't see that in the diff anywhere.

2) Fix or replace "remove superseded upstream data file" so what whatever you do say matches against what I see in each proposed diff (you can say something different for Xenial and Zesty if needed of course). If you're removing dats I think we do need to mention it. Saying something like "source: remove firmware dat files as needed to bring the shipped set in sync with those shipped in Artful" would be fine.

3) In general, make sure that the changelogs accurately describe the diffs that I will review.

Thanks

Robie Basak (racb) wrote :

I've reviewed Dave's updated upload. They look good - thanks!

One note: I think due to an omission debian/changelog for Zesty doesn't mention that it fixes Skylake, whereas the Xenial debian/changelog does state this. The binary blobs shipped between the two are identical, so presumably the changelog descriptions should be identical. I don't think this is important enough to require a further iteration.

description: updated
Changed in intel-microcode (Ubuntu Xenial):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-xenial

Hello C, or anyone else affected,

Accepted intel-microcode into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/intel-microcode/3.20170707.1~ubuntu16.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in intel-microcode (Ubuntu Zesty):
status: Confirmed → Fix Committed
tags: added: verification-needed-zesty
Robie Basak (racb) wrote :

Hello C, or anyone else affected,

Accepted intel-microcode into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/intel-microcode/3.20170707.1~ubuntu17.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Robie Basak (racb) wrote :

Yakkety is EOL now.

Changed in intel-microcode (Ubuntu Yakkety):
status: Confirmed → Won't Fix
Robie Basak (racb) wrote :

Additional SRU verification notes (also in bug description):

SRU verification needs to take care to consider CPUs actually tested. We should have a representative sample of CPUs tested in SRU verification reports before considering release to the updates pockets.

Given the potential severity of regressions, we should keep this in the proposed pockets for longer than the usual minimum ageing period. Let's have users opt-in to this update first, and only recommend it once we confidence that a reasonable number (and representative CPU sample) of opted-in users have not hit any problems.

Testers: please mark verification-done-* only after you consider that the above additional requirements have been met.

Dave Chiluk (chiluk) wrote :

@rbasak, The reason for the omission of "skylake" from the changelog entry for zesty is because zesty already had 3.20170511.1~ubuntu17.04.0 in -updates which contained the fixes for the majority of Skylake processors, but not Kaby Lake. So for zesty this release only fixes kaby lake processors, and maybe a few high end skylake-x processors. Since Xenial did not have that intermediate microcode version, this is the first revision of the package that resolves the issue for both skylake and kabylake processors.

Adrien Beau (adrienbeau) wrote :

Here is a successful test of a Core i5-6600 Skylake CPU under Xenial.

$ uname -a
Linux 4.10.0-28-generic #32~16.04.2-Ubuntu SMP Thu Jul 20 10:19:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ cat /proc/cpuinfo
(...)
vendor_id : GenuineIntel
cpu family : 6
model : 94
model name : Intel(R) Core(TM) i5-6600 CPU @ 3.30GHz
stepping : 3
microcode : 0xa6
(...)

Before installing the intel-microcode package I had:

kernel: microcode: sig=0x506e3, pf=0x2, revision=0xa6
kernel: microcode: Microcode Update Driver: v2.2.

After installing the 3.20151106.1 package version:

kernel: microcode: sig=0x506e3, pf=0x2, revision=0xa6
kernel: microcode: Microcode Update Driver: v2.2.

(No surprise, since the microcode release was older than the CPU.)

After installing the 3.20170707.1~ubuntu16.04.0 package version:

kernel: microcode: microcode updated early to revision 0xba, date = 2017-04-09
kernel: microcode: sig=0x506e3, pf=0x2, revision=0xba
kernel: microcode: Microcode Update Driver: v2.2.

So, the update worked fine, and the machine booted and seems to run fine. :)

I'll pay attention in the next few days if the machine exhibits any weird behavior. It is extremely stable, so any new problem should stick out clearly.

Robie Basak (racb) wrote :

> @rbasak, The reason for the omission of "skylake" from the changelog entry for zesty is because zesty already had 3.20170511.1~ubuntu17.04.0 in -updates which contained the fixes for the majority of Skylake processors, but not Kaby Lake.

Ah. That makes sense. Sorry!

Adrien Beau (adrienbeau) wrote :

Another test, this time on a NUC with a Pentium N3700 Braswell CPU under Xenial.

$ uname -a
Linux 4.10.0-28-generic #32~16.04.2-Ubuntu SMP Thu Jul 20 10:19:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ cat /proc/cpuinfo
(...)vendor_id : GenuineIntel
cpu family : 6
model : 76
model name : Intel(R) Pentium(R) CPU N3700 @ 1.60GHz
stepping : 3
microcode : 0x363
(...)

Before installing the intel-microcode package I had:

kernel: microcode: sig=0x406c3, pf=0x1, revision=0x363
kernel: microcode: Microcode Update Driver: v2.2.

After installing the 3.20151106.1 package, nothing changed.
After installing the 3.20170707.1~ubuntu16.04.0 package, nothing changed.

This is as expected, since this CPU is not affected by the problem, and (as far as I know) has no recent microcode update.

The new package (and the update from the old to the new) works fine on this machine.

asgard2 (kamp000x) wrote :

when can we expect the xenial release, could not take long since the status committed ?

Dimitri John Ledkov (xnox) wrote :

It was committed 3 days ago (the latest upload) and it is usually held in proposed for at least 7 days.

Robie Basak (racb) wrote :

@asgard2

Have you been able to test the proposed update please?

The minimum aging period is indeed 7 days, but I did say that I think this particular update needs wider testing and aging due to the expected severity of any regressions.

Thank you to Adrien for taking the time to test this update.

54 people have marked themselves as being affected by this issue, and over 30 people seem to care enough about this issue to subscribe personally to the bug. So come on, get testing please.

Dave Chiluk (chiluk) wrote :

I have been running this on my kabylake laptop and skylake desktop under X for quite some time now, and haven't hit any issues that I can attribute to the microcode. I attempted to follow the reproducer here https://caml.inria.fr/mantis/view.php?id=7452 , but some of the versions of packages required for that reproducer are too old in the X archive, and I don't want to cruft up my machine. I plan on building a chroot to test that reproducer, but I haven't gotten around to it.

@rbasak, I agree with your assessment to let this soak in -proposed for an extended amount of time in the hopes of getting additional testing.

Paul Menzel (paulmenzel) wrote :

> The minimum aging period is indeed 7 days, but I did say that I think this particular update needs wider testing and aging due to the expected severity of any regressions.

Debian distributed the update already if I am not mistaken. I haven’t heard of any regression. Also I don’t recall anything in the past where updated microcode updates cost regressions. So I don’t see the high caution is justified compared to any other package.

Anyway, for what it’s worth, I tested this on a TUXEDO Book BU1406 with Ubuntu 17.04.

Tested on my system:

Linux sxps 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial

CPU:

Vendor ID: GenuineIntel
CPU family: 6
Model: 78
Model name: Intel(R) Core(TM) i7-6560U CPU @ 2.20GHz
Stepping: 3

Before: (I disabled HT in the BIOS)

Jul 17 22:49:44 sxps kernel: [ 1.220419] microcode: CPU0 sig=0x406e3, pf=0x40, revision=0x9e
Jul 17 22:49:44 sxps kernel: [ 1.220460] microcode: CPU1 sig=0x406e3, pf=0x40, revision=0x9e
Jul 17 22:49:44 sxps kernel: [ 1.220521] microcode: Microcode Update Driver: v2.01 <email address hidden>, Peter Oruba

After (currently running):

[ 1.236290] microcode: CPU0 sig=0x406e3, pf=0x40, revision=0xba
[ 1.236294] microcode: CPU1 sig=0x406e3, pf=0x40, revision=0xba
[ 1.236327] microcode: CPU2 sig=0x406e3, pf=0x40, revision=0xba
[ 1.236351] microcode: CPU3 sig=0x406e3, pf=0x40, revision=0xba
[ 1.236432] microcode: Microcode Update Driver: v2.01 <email address hidden>, Peter Oruba

C de-Avillez (hggdh2) wrote :

running on 17.04; no issues loading the new package:

[ 2.985314] microcode: sig=0x306c3, pf=0x10, revision=0x22
[ 3.013156] microcode: Microcode Update Driver: v2.2.

# on return from suspend:

[336419.388901] microcode: sig=0x306c3, pf=0x10, revision=0x12
[336419.389861] microcode: updated to revision 0x22, date = 2017-01-27

CPU data:

vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz
stepping : 3
microcode : 0x22
cpu MHz : 1783.447
cache size : 6144 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 4

I am not affected by this issue, as I stated when I opened the bug; nevertheless, I can show that the new package did not regress my system.

asgard2 (kamp000x) wrote :

@Robie Basak

I updated the intel-microcode package ("3.20170707.1~ubuntu16.04.0") from proposed.
This version is running fine so far on the "Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz"

Dave Chiluk (chiluk) wrote :

Marking xenial-verification-done

I was able to reproduce the compilation crash as described on the ocaml bug in about 30 minutes on my Skylake machine. After upgrading firmware, I have been running for simultanous compilations loops for the last 2 hours with no crash. I will let them continue to run although I don't expect to see any issues.

<Test-case>
Download report.tar.gz from https://caml.inria.fr/mantis/view.php?id=7452 and place in your schroot scratch directory.
$ mk-sbuild artful --arch=amd64
$ schroot -c artful -u root
// Artful was chosen as it contains the required versions of Ocaml for the reproducer.
$ apt install ocaml opam ocaml-findlib m4
$ opam init
$ opam install extprot
$ eval `opam config env`
$ while ocamlfind opt -c -g -bin-annot -ccopt -g -ccopt -O2 -ccopt -Wextra -ccopt '-Wstrict-overflow=5' -thread -w +a-4-40..42-44-45-48-58 -w -27-32 -package extprot test.ml -o test.cmx; do echo "ok"; done
</test-case>

// forgive the test-case for incorrect ocaml-isms, but as I've never developed using it...

description: updated
tags: added: verification-done-xenial
removed: verification-needed-xenial
Dave Chiluk (chiluk) wrote :

I have now killed my 4-simultaneous compilation loop tests after 20 hours. With the previous microcode, 2 of 3 loop threads died with a segfault within 1 hour, so I consider this fixed.

@Others, Has anyone tested this with skylake/kabylake on zesty?
I have installed and boot tested using Ivy-bridge on zesty, but I'd feel slightly better if someone could test Sky/Kaby on zesty explicitly. Bonus points if you recreate the crash scenario, and verify resolution.

I don't think we need any more positive tests on X, please only report errors or failures going forward for X.

Dave Chiluk (chiluk) wrote :

I just verified skylake + 20170707 on zesty. I did not rerun the testcase, but I did do boot testing.

Marking verification-done.

tags: added: verification-done verification-done-zesty
removed: verification-needed verification-needed-zesty
Changed in intel-microcode (Ubuntu Yakkety):
assignee: Dave Chiluk (chiluk) → nobody
Robie Basak (racb) wrote :

> So I don’t see the high caution is justified compared to any other package.

Because: a) Unlike regular software updates, testing on one CPU model/stepping doesn't test on any other CPU model/stepping; and b) if the updates does regress, unlike most other updates this may render systems unbootable, meaning that fixing any regression isn't as simple as just releasing another update.

Thanks for the testing so far everyone. Here's mine on Xenial:

robie@mal:~$ dpkg-query -W intel-microcode
intel-microcode 3.20170707.1~ubuntu16.04.0
robie@mal:~$ grep -E 'model|stepping' /proc/cpuinfo | sort -u
model : 94
model name : Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
stepping : 3
robie@mal:~$ journalctl -k | grep microcode
Aug 03 13:35:16 mal kernel: microcode: CPU0 microcode updated early to revision 0xba, date = 2017-04-09
Aug 03 13:35:16 mal kernel: microcode: CPU1 microcode updated early to revision 0xba, date = 2017-04-09
Aug 03 13:35:16 mal kernel: microcode: CPU2 microcode updated early to revision 0xba, date = 2017-04-09
Aug 03 13:35:16 mal kernel: microcode: CPU3 microcode updated early to revision 0xba, date = 2017-04-09
Aug 03 13:35:16 mal kernel: microcode: CPU0 sig=0x506e3, pf=0x2, revision=0xba
Aug 03 13:35:16 mal kernel: microcode: CPU1 sig=0x506e3, pf=0x2, revision=0xba
Aug 03 13:35:16 mal kernel: microcode: CPU2 sig=0x506e3, pf=0x2, revision=0xba
Aug 03 13:35:16 mal kernel: microcode: CPU3 sig=0x506e3, pf=0x2, revision=0xba
Aug 03 13:35:16 mal kernel: microcode: CPU4 sig=0x506e3, pf=0x2, revision=0xba
Aug 03 13:35:16 mal kernel: microcode: CPU5 sig=0x506e3, pf=0x2, revision=0xba
Aug 03 13:35:16 mal kernel: microcode: CPU6 sig=0x506e3, pf=0x2, revision=0xba
Aug 03 13:35:16 mal kernel: microcode: CPU7 sig=0x506e3, pf=0x2, revision=0xba
Aug 03 13:35:16 mal kernel: microcode: Microcode Update Driver: v2.01 <email address hidden>, Peter Oruba
robie@mal:~$ dpkg-query -W intel-microcode
intel-microcode 3.20170707.1~ubuntu16.04.0

Actually, it is not hard to bypass a boot-killer microcode update issue. Just add the "dis_ucode_ldr" parameter to the kernel command line.

To make it trivial, add that to a "safe mode" grub menu entry...

Robie Basak (racb) wrote :

> Just add the "dis_ucode_ldr" parameter to the kernel command line.

That's hardly a "just". Remember that Ubuntu is for everyone, not just people who know how to tweak their bootloaders, or even understand how to get to the grub boot menu. It's unreasonable for all Ubuntu desktop users everywhere to have to jump through these hoops. Let's try and avoid it happening.

Simon Déziel (sdeziel) wrote :

Worked fine for me with Xenial on a laptop:

$ dpkg-query -W intel-microcode
intel-microcode 3.20170707.1~ubuntu16.04.0

$ grep -E 'model|stepping' /proc/cpuinfo | sort -u
model : 142
model name : Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
stepping : 9

$ journalctl -k | grep microcode
Aug 17 14:17:26 simon-laptop kernel: microcode: CPU0 microcode updated early to revision 0x62, date = 2017-04-27
Aug 17 14:17:26 simon-laptop kernel: microcode: CPU1 microcode updated early to revision 0x62, date = 2017-04-27
Aug 17 14:17:26 simon-laptop kernel: microcode: CPU0 sig=0x806e9, pf=0x80, revision=0x62
Aug 17 14:17:26 simon-laptop kernel: microcode: CPU1 sig=0x806e9, pf=0x80, revision=0x62
Aug 17 14:17:26 simon-laptop kernel: microcode: CPU2 sig=0x806e9, pf=0x80, revision=0x62
Aug 17 14:17:26 simon-laptop kernel: microcode: CPU3 sig=0x806e9, pf=0x80, revision=0x62
Aug 17 14:17:26 simon-laptop kernel: microcode: Microcode Update Driver: v2.01 <email address hidden>, Peter Oruba

Simon Déziel (sdeziel) wrote :

On recent machine (88 cores + HT) running Xenial:

$ dpkg-query -W intel-microcode
intel-microcode 3.20170707.1~ubuntu16.04.0

$ grep -E 'model|stepping' /proc/cpuinfo | sort -u
model : 79
model name : Intel(R) Xeon(R) CPU E5-4669 v4 @ 2.20GHz
stepping : 1

$ journalctl -k | grep microcode
Aug 11 09:39:18 node14.mgmt.hre.local kernel: microcode: CPU0 sig=0x406f1, pf=0x20, revision=0xb00001e
...
Aug 11 09:39:18 node14.mgmt.hre.local kernel: microcode: CPU175 sig=0x406f1, pf=0x20, revision=0xb00001e
Aug 11 09:39:18 node14.mgmt.hre.local kernel: microcode: Microcode Update Driver: v2.01 <email address hidden>, Peter Oruba

It's probably not that relevant but I also couldn't find any regression on a very old Core 2 Duo (no new microcode) running Xenial:

$ dpkg-query -W intel-microcode
intel-microcode 3.20170707.1~ubuntu16.04.0

$ grep -E 'model|stepping' /proc/cpuinfo | sort -u
model : 23
model name : Intel(R) Xeon(R) CPU E3110 @ 3.00GHz
stepping : 6

$ journalctl -k | grep microcode
Aug 17 14:29:28 xeon kernel: microcode: CPU0 microcode updated early to revision 0x60f, date = 2010-09-29
Aug 17 14:29:28 xeon kernel: microcode: CPU1 microcode updated early to revision 0x60f, date = 2010-09-29
Aug 17 14:29:28 xeon kernel: microcode: CPU0 sig=0x10676, pf=0x1, revision=0x60f
Aug 17 14:29:28 xeon kernel: microcode: CPU1 sig=0x10676, pf=0x1, revision=0x60f
Aug 17 14:29:28 xeon kernel: microcode: Microcode Update Driver: v2.01 <email address hidden>, Peter Oruba

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.