intel-microcode 3.20180312.0 causes lockup at login screen

[Erich E. Hoover]

[Impact]

 * Some systems experience kernel lockups after updating to the latest intel-microcode
   package or when receiving updated microcode from a BIOS update.

 * In many cases, the lockups occur before users can reach the login screen which makes
   it very difficult to debug/workaround.

[Test Case]

 * The most reliable test case currently known is to install the sssd package. Lockups
   may occur during package installation (disable IBPB by writing 0 to
   /proc/sys/kernel/ibpb_enabled to prevent this from happening). A lockup will most
   likely occur just after booting the system up as the lock screen is displayed.

[Regression Potential]

 * The fix is in the task switching code of the kernel so complexity of the change is
   relatively high.

[Original Report]

I don't know if this is a problem with the kernel or the microcode, but we have a significant number of computers in our organization (on both 16.04 and 17.10) that fail if they have both updated. Booting with either linux-image-4.13.0-36-generic or intel-microcode 3.20180108.0+really20170707ubuntu17.10.1 allows all these computers to boot.

## Workaround ##
1. Boot the system with the dis_ucode_ldr kernel boot parameter to temporary avoid the problem:
   https://wiki.ubuntu.com/Kernel/KernelBootParameters
2. Install the previous version of package from
   https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+build/14261530/+files/intel-microcode_3.20180108.0+really20170707ubuntu16.04.1_amd64.deb
3. (Optional) Hold the package so that it won't be upgraded accidentally
   sudo apt-mark hold intel-microcode

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1759920

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Tyler Hicks (tyhicks)]

Thanks for the report, Erich!

Does the lockup happen as soon as you reach the login screen or does it happen once you put in your password and hit enter?

Do you happen to use NVIDIA graphics drivers on the system that's locking up?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jeremy Gebben (jgebben)]

ApportVersion: 2.20.7-0ubuntu3.7
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 17.10
NonfreeKernelModules: nvidia_uvm nvidia_drm nvidia_modeset nvidia
Package: linux
PackageArchitecture: amd64
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
 TERM=screen-256color
 XDG_RUNTIME_DIR=<set>
 PATH=(custom, no user)
ProcVersionSignature: Ubuntu 4.13.0-36.40-generic 4.13.13
Tags: artful
Uname: Linux 4.13.0-36-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: Domain Users adm cdrom dialout dip lpadmin plugdev sambashare sbuild sudo wireshark
_MarkForUpload: True

[Jeremy Gebben (jgebben)]

apport information

[Jeremy Gebben (jgebben)]

apport information

[Jeremy Gebben (jgebben)]

apport information

[Tyler Hicks (tyhicks)]

Erich, could you include the output of 'iucode-tool --scan-system'?

[Erich E. Hoover (ehoover)]

> Does the lockup happen as soon as you reach the login screen or does it happen once you put in your password and hit enter?

It sometimes happens as soon as the login screen shows up, sometimes it happens after you hit enter, and roughly 1:8 tries it will make it all the way to the desktop.

> Do you happen to use NVIDIA graphics drivers on the system that's locking up?

I believe that all of our systems are running NVIDIA drivers, but we have replicated it on a system not running the NVIDIA driver.

> Erich, could you include the output of 'iucode-tool --scan-system'?

Most of them report:
iucode-tool: system has processor(s) with signature 0x000906e9
We have one that reports:
iucode-tool: system has processor(s) with signature 0x000906ea

[Kyle Etsler (ketsler)]

I have replicated this issue on several systems. I have had the lockup occur at several moments at the lock screen. It usually locks up once you reach the login screen, sometimes when you start typing your password, sometimes when you hit enter after typing in a correct password.

I have replicated this issue on systems that have NVIDIA GPUs/Drivers, and those without.

The systems I have had this issue on have had 6th or 7th generation i7 Intel processors and all are laptops. I have a desktop with 8th generation Intel, with Linux Kernel 4.13.0-37 and the updated Intel microcode and I am not having this issue.

[Tyler Hicks (tyhicks)]

Could someone else with an affected system verify Erich's report that booting into the linux-image-4.13.0-36-generic kernel while using intel-microcode 3.20180312.0~ubuntu17.10.1 results in a system that does not experience lock ups?

[Eric Chandler (echandler)]

ApportVersion: 2.20.1-0ubuntu2.15
Architecture: amd64
CurrentDesktop: Unity
DistroRelease: Ubuntu 16.04
NonfreeKernelModules: nvidia_uvm nvidia_drm nvidia_modeset nvidia
Package: linux
PackageArchitecture: amd64
ProcVersionSignature: Ubuntu 4.13.0-37.42~16.04.1-generic 4.13.13
Tags: xenial
Uname: Linux 4.13.0-37-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: Domain Users adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True

[Eric Chandler (echandler)]

apport information

[Eric Chandler (echandler)]

apport information

[Eric Chandler (echandler)]

apport information

[Eric Chandler (echandler)]

apport information

[Eric Chandler (echandler)]

Booting with 4.13.0-36-generic prevented computer lockup during startup.

[John Bryant (bryantj247)]

I'm having the same problem. 4.13.0-36-generic works, 4.13.0-37-generic freezes up before I can log in. I have a 2nd generation core i7 and AMD graphics.

[Tyler Hicks (tyhicks)]

John, could you run 'apport-collect 1759920' or, alternatively, include your /proc/cpuinfo contents and the 'iucode-tool --scan-system' output?

[John Bryant (bryantj247)]

ApportVersion: 2.20.7-0ubuntu3.7
Architecture: amd64
CurrentDesktop: GNOME
DistroRelease: Ubuntu 17.10
InstallationDate: Installed on 2018-02-17 (40 days ago)
InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64 (20180105.1)
Package: linux
PackageArchitecture: amd64
ProcVersionSignature: Ubuntu 4.13.0-36.40-generic 4.13.13
Tags: artful
Uname: Linux 4.13.0-36-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip libvirt lpadmin plugdev sambashare sudo
_MarkForUpload: True

[John Bryant (bryantj247)]

apport information

[John Bryant (bryantj247)]

apport information

[John Bryant (bryantj247)]

apport information

[John Bryant (bryantj247)]

apport information

[林博仁(Buo-ren, Lin) (buo-ren-lin)]

ApportVersion: 2.20.1-0ubuntu2.15
Architecture: amd64
CurrentDesktop: KDE
DistroRelease: KDE 16.04
InstallationDate: Installed on 2017-04-11 (352 days ago)
InstallationMedia: neon user "Xenial" - Build amd64 LIVE Binary 20170323-10:36
Package: linux
PackageArchitecture: amd64
ProcVersionSignature: Ubuntu 4.13.0-37.42~16.04.1-generic 4.13.13
Tags: third-party-packages xenial
Uname: Linux 4.13.0-37-generic x86_64
UnreportableReason: 這不是官方的 KDE 套件。請移除任何第三方套件並重試。
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm bumblebee cdrom dip docker libvirtd lpadmin lxd plugdev sambashare scanner sudo ubridge vboxusers video
_MarkForUpload: True

[林博仁(Buo-ren, Lin) (buo-ren-lin)]

apport information

[林博仁(Buo-ren, Lin) (buo-ren-lin)]

apport information

[林博仁(Buo-ren, Lin) (buo-ren-lin)]

apport information

[林博仁(Buo-ren, Lin) (buo-ren-lin)]

apport information

[林博仁(Buo-ren, Lin) (buo-ren-lin)]

I have this issue as well, and I'm pretty sure this is a regression of the latest released intel-microcode package as I deliberately upgrade this single one just today.

[林博仁(Buo-ren, Lin) (buo-ren-lin)]

This is a Ivybridge based laptop, ASUS S550CB to be specific.

[Gordon Lack (gordon-lack)]

Bug #1760095 (mine) reports it for two 17.10 systems (Intel graphics only) both on 4.13.0-37-generic.
An 18.04beta system 4.15.0-13-generic with the "same" microcode version is OK.

[Erich E. Hoover (ehoover)]

I've identified a couple more systems that are affected by this issue:
iucode-tool: system has processor(s) with signature 0x00040651
iucode-tool: system has processor(s) with signature 0x000806e9

That second one was actually having trouble with 4.4.0-116-generic (boots fine with 4.4.0-112-generic). I installed the HWE package on the system and it will not work with either 4.13.0-37-generic or 4.13.0-36-generic, but it will work on 4.13.0-37-generic with intel-microcode 3.20180108.0+really20170707ubuntu16.04.1.

[Hannu Koivisto (azure)]

Kubuntu 17.10, kernel 4.13.0-37-generic, Dell E6330 with Intel graphics (iucode-tool: system has processor(s) with signature 0x000306a9, cpuinfo attached), after intel-microcode upgrade to 3.20180312.0~ubuntu17.10.1 => lockup during boot after entering disk encryption password but before login screen appears.

Boot with 4.13.0-36-generic => no problem.

[Tyler Hicks (tyhicks)]

I'm still unable to reproduce these lock ups and I've tested on three different systems. So, I'm reliant on you (very helpful) bug reporters.

Could someone test out the 4.13.0-38-generic kernel from artful-proposed to see if you experience any lock ups with the 3.20180312.0 microcode?

In the meantime, I'm looking through the changes between 4.13.0-36 and 4.13.0-37 to see if there's anything that sticks out. Nearly all of the changes are ARM related so that limits the possibilities in a nice way.

[Gordon Lack (gordon-lack)]

> Could someone test out the 4.13.0-38-generic kernel from artful-proposed
> to see if you experience any lock ups with the 3.20180312.0 microcode?

Any simple install instructions?

[Tyler Hicks (tyhicks)]

On 03/30/2018 10:59 AM, Gordon Lack wrote:
>> Could someone test out the 4.13.0-38-generic kernel from artful-proposed
>> to see if you experience any lock ups with the 3.20180312.0 microcode?
>
> Any simple install instructions?

Here are the typical install instructions for enabling proposed:

  https://wiki.ubuntu.com/Testing/EnableProposed

Be sure to follow the "Selective upgrading from -proposed" section.

[Gordon Lack (gordon-lack)]

For me, on my laptop (Intel(R) Core(TM) i5-4330M CPU @ 2.80GHz) with Kubuntu 17.10, the 4.13.0-38-generic kernel (signed image) has the same problem.
In fact it was slightly worse, a it seems to have wiped out my Ubuntu UEFI booting (boots straight to Windows - I can get to the Ubuntu menu from there and that works).
Now fixing that...

Reverting to the older intel-microcode with 4.13.0-38-generic is OK.

[John Bryant (bryantj247)]

It behaves a little differently. Before, it would clear the screen and get stuck. Now it shows this:

[ 36.067981] watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [kworker/5:1:71]
[ 36.067983] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [thermald:1113]
[ 36.527766] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0
[ 37.482793] NMI watchdog: Watchdog detected hard LOCKUP on cpu 3
[ 41.598740] NMI watchdog: Watchdog detected hard LOCKUP on cpu 1

Then there's a couple repeats of the first 2 lines at later times.

[camypaj (mantonijevic)]

Hi all,

I'm experiencing the same issue on Linux Mint 18.2 (based on ubuntu 16.04), running 4.13.0-37-generic. I'm using Luks, so after decrypting my drive(s), the system continues to boot, but never reaches the login screen.

iucode-tool: system has processor(s) with signature 0x000406e3
Cpu: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
inxi -G:
Graphics: Card: Intel Sky Lake Integrated Graphics
           Display Server: X.org 1.18.4 drivers: (unloaded: fbdev,vesa)

Additionally, both kernel versions (linux-image-4.13.0-37-generic and linux-image-4.13.0-36-generic) had the same issue, which went away only after I removed intel-microcode (3.20180312.0~ubuntu16.04.1). Needless to say, it worked perfectly fine with the old version from xenial (3.20151106.1).

Any other tests I could run? Would it help if I tested 4.13.0-38-generic?

Thanks & regards,
Camypaj

[camypaj (mantonijevic)]

Sorry for spamming, but I could not find how to edit my previous comment - in my case the freeze happened with every boot. Additionally, for some reason, I was not able to boot KDE Neon from a usb either, and that one was not used or updated in a while. I was thinking that it's somehow swap-related, but it wasn't, since the problem persisted after I commented out the swap in fstab.

[Gordon Lack (gordon-lack)]

FYI: These are the signatures for my three systems.

Failing for these on 17.10 (4.13.0-37-generic):
    signature 0x000306c3 (and for 4.13.0-38-generic)
    signature 0x000206a7

OK for this on 18.04beta (4.15.0-13-generic):
    signature 0x000306d4

[John Bryant (bryantj247)]

I just found something interesting. Installing or removing intel-microcode only runs update-initramfs for the latest kernel, which is why my 4.13.0-36-generic kept working. When I ran
"sudo update-initramfs -u -k 4.13.0-36-generic", that kernel stopped working too.

[Erich E. Hoover (ehoover)]

That's very interesting John, that could also explain why we had a system:
1) get updated
2) fail on 4.4.0-116-generic
3) work on 4.4.0-112-generic
4) install 4.13.0-36-generic and 4.13.0-37-generic
5) fail on 4.13.0-36-generic and 4.13.0-37-generic
6) get intel-microcode downgraded to 3.20180108.0+really20170707ubuntu16.04.1
7) work on 4.13.0-37-generic

[Gordon Lack (gordon-lack)]

>> Installing or removing intel-microcode only runs update-initramfs for the latest kernel

I eventually noticed that too.
So it means that the problem follows the microcode - nothing to do with the kernel version.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Justa Guy (nginus)]

>> Gordon Lack (gordon-lack) wrote 23 hours ago:

>> >> Installing or removing intel-microcode only runs update-initramfs for the latest kernel

>> I eventually noticed that too.
>> So it means that the problem follows the microcode - nothing to do with the kernel version.

Well, 4.13.0-32 seems to be unaffected...

On my Dell Inspiron 5559 laptop, with Ubuntu 16.04.3 installed, I had been running linux-signed-image-4.13.0-36-generic, then it wouldn't boot after installing Thursday's intel-microcode update. It would however boot into linux-signed-image-4.13.0-32-generic, after which I tried installing linux-signed-image-4.13.0-37-generic to see if that would solve the problem, but it didn't. As it stands I can only boot using the linux-signed-image-4.13.0-32-generic kernel option in grub manually.

In both cases, the actual boot process varied in how far it would get before failing.

The first time, trying linux-signed-image-4.13.0-36-generic, the console text with the green <OK> column on the left proceeded to after loading what appeared as completely, then remained stuck with a steady (not flashing) grey cursor in screen position 0,0. I waited maybe 4 minutes for it to load the desktop, but no. As opposed to when booting into linux-signed-image-4.13.0-32-generic, the mouse pointer appears approximately 5 seconds after the point where it would hang, with a whole desktop following maybe 30 seconds afterward.

The first time I tried the linux-signed-image-4.13.0-37-generic kernel, it wouldn't proceed past the loading bluetooth part of the console text with the green <OK> column to the left. The reboot after this seemed to get just past the part with the cursor stuck at 0,0, in this case with it disappearing- I thought a mouse pointer was just about to appear, but it stopped. Waited only a minute or 2 here before rebooting again. The third try at linux-signed-image-4.13.0-37-generic got stuck where linux-signed-image-4.13.0-36-generic had previously, with the cursor at 0,0.

Not sure about the official workaround as is posted in the bug description, as I've been using linux-signed-image-4.13.0-32-generic since. I don't know why this one works, or if this is a safe alternative.

[Gordon Lack (gordon-lack)]

>> Well, 4.13.0-32 seems to be unaffected...

Only because it isn't the latest kernel you have, so it has never had the latest microcode added into it - it will still be loading the previous (working) version.

[Gordon Lack (gordon-lack)]

This is a similar thread on Arch Linux:
    https://bugs.archlinux.org/task/57511

Includes the comment:
   "Wasn't the 20180108 microcode update retracted because of issues? Maybe we should downgrade back to 20171117."

There is no 20180108 download available at Intel now:
    https://downloadcenter.intel.com/download/27591/Linux-Processor-Microcode-Data-File?product=84984

Looks like the latest one is messed up too - at least for some processors.

[Erich E. Hoover (ehoover)]

By any chance, do all of you have sssd installed? One of my colleagues (Kyle Etsler) has established a relationship between this problem and having that package installed. I've verified this on my own machine now, and it makes some sense given that sssd is involved in the login process.

[Gordon Lack (gordon-lack)]

> By any chance, do all of you have sssd installed?

No. I don't have it installed on any of my systems, two of which are affected.

[kolya (mar-kolya)]

It looks like this is the same problem as https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1746418. And adding 'noibpb' kernel boot parameter helps, at least in my case.

[Gordon Lack (gordon-lack)]

>> And adding 'noibpb' kernel boot parameter helps, at least in my case.

However, according to:
   https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown/MitigationControls
doing that,
"Disabling these features removes mitigations for CVE-2017-5715 (aka Spectre / Variant 2).", which sort-of defeats what the microcode fix is for.

However, the rest of the thread for Bug #1746418 does explain why my 18.04beta system is OK - it's running 4.15.0-xxx

[Carl van Schaik (navlrac)]

Affects me on:
Intel(R) Xeon(R) CPU E3-1240 v5 @ 3.50GHz (family: 0x6, model: 0x5e, stepping: 0x3)

uninstall / disable intel-microcode has restored service on my DL20 G9 server.

Kernel cmdline dis_ucode_ldr worked to allow me to boot and uninstall the package.

> By any chance, do all of you have sssd installed?
This server has sssd, but unclear if its related.

[Antti Hukkanen (ahukkanen)]

I can confirm this with the following processor:
vendor_id : GenuineIntel
cpu family : 6
model : 58
model name : Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz

iucode-tool reports the following signature:
iucode-tool: system has processor(s) with signature 0x000306a9

This signature matches with ehoover's (#11) and I had exactly the same error on startup as bryantj247 (#41):

NMI watchdog: Watchdog detected hard LOCKUP on cpu 0
NMI watchdog: Watchdog detected hard LOCKUP on cpu 1
NMI watchdog: Watchdog detected hard LOCKUP on cpu 2
NMI watchdog: Watchdog detected hard LOCKUP on cpu 3

This machine has sssd installed but unsure if it is related. This error happened very early in the bootup process.

[Jan Claeys (janc)]

I had this same problem on a system with an "ASUS H170M-PLUS" motherboard, a Core i5-6600 CPU, internal graphics and up-to-date Ubuntu 17.10.

Booting into the recovery console worked, but when trying to boot into the desktop it would always hang somewhere before GDM becomes visible.

After installing a recent UEFI firmware upgrade (to version 3606) the system booted normally again, and the kernel didn't try to load any microcode after that; based on the revision number, it seems like it contained the same microcode fix as the 'intel-microcode' (but also a ME firmware upgrade, and likely also some other necessary fixes).

I wonder if the microcode fix also requires changes to the ME and/or to some UEFI functions in order to work properly?

[Tyler Hicks (tyhicks)]

I think that I've figured out this lockup thanks to some information in duplicate bugs. Specifically, this comment:

https://bugs.launchpad.net/ubuntu/+source/intel-microcode/+bug/1760264/comments/7

as well as the nice git bisect work of jsalisbury that I recently discovered in bug 1746418. This is the problematic commit:

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/artful/commit/?id=96d520d

When the kernel does a task switch due to a task that was confined by AppArmor exiting, the task's pi_lock is taken in the exit() path and then switch_mm() is calling ___ptrace_may_access() which then calls down into apparmor which then calls into audit if the old task can't ptrace the new task. Eventually, the audit subsystem tries to take the pi_lock again when waking up the task.

This would be a little easier to be sure of with a nice lockdep warning from a debug kernel build but I'm fairly sure this is what's going on. I suspect that swapping out the commit above with this upstream commit will fix the problem:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=18bf3c3ea

It doesn't call into AppArmor to see if IBPB should be used when switching tasks. I'll build a test kernel for affected folks to test with. In the meantime, if someone affected wanted to boot with the apparmor=0 kernel command line option (with the latest artful kernel, without the noibpb kernel command line option, and with the latest intel-microcode package), I'd really appreciate it.

[Mike Pontillo (mpontillo)]

@tyhicks, I don't have an Artful test system, but I confirmed on Xenial (4.4.0-116-generic) that `apparmor=0` is a workaround, without the `noibpb` option and with intel-microcode version 3.20180312.0~ubuntu16.04.1.

[Gordon Lack (gordon-lack)]

> if someone affected wanted to boot with the apparmor=0 kernel command line option (with the latest artful kernel, without the noibpb kernel command line option, and with the latest intel-microcode package), I'd really appreciate it.

Just done this:

Downloaded latest microcode (3.20180312).
Replaced /lib/firmware/intel-ucode with one from the download.
Ran update-initramfs -u -k 4.13.0-38-generic
Rebooted - specifically selected 4.13.0-38-generic to reboot.
Added apparmor=0 as a boot option.
Booted.

=> All looks OK. System still up and running after 5 mins.

Rebooted to 4.13.0-38-generic without specifying apparmor=0.

=> System locks up at the "pulsing" kubuntu logo.

Rebooted back to 4.13.0-38-generic with apparmor=0 (and
fsck.mode=force) to switch back to the previous microcode on
4.13.0-38-generic.

[Gordon Lack (gordon-lack)]

I've since left the 4.13.0-38-generic kernel + 3.20180312 microcode + apparmor=0 option running for >30 mins (top constantly running plus a short compilation using all 4 cores in parallel, make -j4, of a github download).

It still looked OK.

[John Bryant (bryantj247)]

It works so far with apparmor=0, and jams without it. I'll keep running it for a while to see if it stays stable.

I used a slightly different method of getting the latest microcode:
sudo apt-mark unhold intel-microcode
sudo apt upgrade

This ran update-initramfs on 4.13.0-38, so everything should be up to date.

Also, I don't have sssd installed.

[John Bryant (bryantj247)]

It was working fine for a few minutes. I opened Firefox, played some Hulu, opened Android Studio, connected my tablet, and all was well. When I tried to allow USB debugging on the tablet, the system locked up.

I had been using 4.13.0-37 prior to trying the new microcode again, so I'll revert the microcode and try the same sequence of events with the newer kernel.

[Tyler Hicks (tyhicks)]

Test kernels with the fix are available here:

http://people.canonical.com/~tyhicks/lp1759920/

You most likely only need linux-image-4.13.0-38-generic_4.13.0-38.43+lp1759920.1_amd64.deb and linux-image-extra-4.13.0-38-generic_4.13.0-38.43+lp1759920.1_amd64.deb

You should test that kernel without the noibpb and without the apparmor=0 kernel command line options, and with the latest intel-microcode package. When replying with testing feedback, please include output from the following commands:

$ cat /proc/version_signature
Ubuntu 4.13.0-38.43+lp1759920.1-generic 4.13.16
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.13.0-38-generic root=UUID=c80800af-5af4-4235-9181-c366ef91c246 ro quiet splash vt.handoff=1
$ dmesg -t | grep -i microcode
microcode: microcode updated early to revision 0xc2, date = 2017-11-16
microcode: sig=0x406e3, pf=0x80, revision=0xc2
microcode: Microcode Update Driver: v2.2.

Thanks again for all of your help. This has been a tough one to track down and it wouldn't have been possible without your help!

[Diego Remolina (dijuremo)]

dijuremo@localhost:~$ cat /proc/version_signature
Ubuntu 4.13.0-38.43+lp1759920.1-generic 4.13.16
dijuremo@localhost:~$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.13.0-38-generic root=/dev/mapper/aevg-root ro quiet splash vt.handoff=7
dijuremo@localhost:~$ dmesg -t | grep -i microcode
microcode: sig=0x306e4, pf=0x1, revision=0x42c
microcode: Microcode Update Driver: v2.2.

Also ran:
https://raw.githubusercontent.com/speed47/spectre-meltdown-checker/master/spectre-meltdown-checker.sh

Spectre and Meltdown mitigation detection tool v0.36+

Checking for vulnerabilities on current system
Kernel is Linux 4.13.0-38-generic #43+lp1759920.1 SMP Tue Apr 3 22:59:23 UTC 2018 x86_64
CPU is Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz

Hardware check
* Hardware support (CPU microcode) for mitigation techniques
  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available: YES
    * CPU indicates IBRS capability: YES (SPEC_CTRL feature bit)
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available: YES
    * CPU indicates IBPB capability: YES (SPEC_CTRL feature bit)
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available: YES
    * CPU indicates STIBP capability: YES
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability: NO
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: NO
  * CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO): NO
  * CPU microcode is known to cause stability problems: NO (model 62 stepping 4 ucode 0x42c)
* CPU vulnerability to the three speculative execution attack variants
  * Vulnerable to Variant 1: YES
  * Vulnerable to Variant 2: YES
  * Vulnerable to Variant 3: YES

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Mitigated according to the /sys interface: YES (kernel confirms that the mitigation is active)
* Kernel has array_index_mask_nospec (x86): NO
* Kernel has the Red Hat/Ubuntu patch: YES
* Kernel has mask_nospec64 (arm): NO
> STATUS: NOT VULNERABLE (Mitigation: OSB (observable speculation barrier, Intel v6))

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigated according to the /sys interface: YES (kernel confirms that the mitigation is active)
* Mitigation 1
  * Kernel is compiled with IBRS/IBPB support: YES
  * Currently enabled features
    * IBRS enabled for Kernel space: NO
    * IBRS enabled for User space: NO
    * IBPB enabled: YES
* Mitigation 2
  * Kernel has branch predictor hardening (arm): NO
  * Kernel compiled with retpoline option: YES
  * Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation)
> STATUS: NOT VULNERABLE (Mitigation: Full generic retpoline, IBPB (Intel v4))

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Mitigated according to the /sys interface: YES (kernel confirms that the mitigation is active)
* Kernel supports Page Table Isolation (PTI): YES (found 'CONFIG_PAGE_TABLE_ISOLATION=y')
* PTI enabled and active: YES
* Running as a Xen PV DomU: NO
> STATUS: NOT VULNERABLE (Mitigation: PTI)

A false sense of secu...

[Gordon Lack (gordon-lack)]

>> Also ran:
>> https://raw.githubusercontent.com/speed47/spectre-meltdown-checker/master/spectre-meltdown-checker.sh

FWIW: I just ran that on my system with the *old* microcode, and it reports "NOT VULNERABLE" for all three variants too.

[Diego Remolina (dijuremo)]

@gordon-lack

>> FWIW: I just ran that on my system with the *old* microcode, and it

Define *old* microcode

Microcode can also come from the latest BIOS if you had updated. So if you really wanted to be sure you have the original CPU microcode which should be vulnerable, you will need to:

1. remove the intel-microcode package
2. rebuild the initrd for the kernel you want to run because the microcode is passed to the kernel
3. Make sure you are on an older BIOS release that did not include Intel microcode updates.

The *older* Ubuntu intel-microcode from January, may contain the first Intel patches, so that is why you want to fully remove the package and rebuilt initrd so that you get the microcode directly from CPU or BIOS.

[Gordon Lack (gordon-lack)]

> Define *old* microcode

The microcode from 3.20180108.0+really20170707ubuntu17.10.1:
 [parent]: dmesg -t | grep -i microcode
 microcode: microcode updated early to revision 0x29, date = 2013-06-12
 microcode: sig=0x206a7, pf=0x2, revision=0x29
 microcode: Microcode Update Driver: v2.2.

If Intel were fixing this in microcode in 2013 then there's something about which they haven't been letting on. This is an Intel(R) Core(TM) i5-2500K(family 6, model 42, stepping 7).

Similarly running this on my laptop (Intel(R) Core(TM) i5-4330M with microcode from that same deb package (microcode revision 0x22, date = 2017-01-27) reports that it is not vulnerable.

Both of these systems hang using the new ("current") microcode (their reports only differs in the CPU type).

One difference I can see on the report for my system running 18.04beta which is OK with the new microcode is that it says NO "Kernel has the Red Hat/Ubuntu patch", but the two systems above, which both suffer with the new microcode, have YES.

[Rob Johnson (rob.johnson)]

@tyhicks

Gargravarr from #ubuntu-kernel :)

Tested your -38 kernel for this issue from comment #67 and I can confirm it allows me to log in with SSSD and an LDAP user. Outputs for your requested commands:

1)
Ubuntu 4.13.0-38.43+lp1759920.1-generic 4.13.16

2)
BOOT_IMAGE=/vmlinuz-4.13.0-38-generic root=/dev/mapper/ubuntu--vg-root ro

3)
microcode: sig=0x806e9, pf=0x80, revision=0x7c
microcode: Microcode Update Driver: v2.2.

Been experiencing this on a range of Dell laptops, mostly XPS 13's with Kaby Lake i7's (7560U) and a Precision M3520 with a 7820HQ.

Look forward to this going into mainline!

Many thanks everyone for the hard work discovering the root cause of this bug, it's a really mind-bending one!

[Gordon Lack (gordon-lack)]

Well, I have a problem testing that kernel.

I've downloaded the linux-image and linux-image extra debs and installed them with dpkg -i.

Rebooting (with the new microcode) hangs.
But on rebooting and checking /proc/version_signature I see:
 Ubuntu 4.13.0-38.43-generic 4.13.16
which makes no sense - my files in /boot are from the new kernel deb (I checked the md5sums).

[Gordon Lack (gordon-lack)]

These are the packages I have installed:

root@gmllaptop:/local/configs/packages# dpkg-query -l '*4.13.0-38*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii linux-headers-4.13.0-38 4.13.0-38.43 all Header files related to Linux kernel version 4.13.0
ii linux-headers-4.13.0-38-generic 4.13.0-38.43 amd64 Linux kernel headers for version 4.13.0 on 64 bit x86 SMP
ii linux-image-4.13.0-38-generic 4.13.0-38.43+lp1759920.1 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
ii linux-image-extra-4.13.0-38-generic 4.13.0-38.43+lp1759920.1 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
ii linux-signed-image-4.13.0-38-generic 4.13.0-38.43 amd64 Signed kernel image generic

The signed one doesn't contain any binaries, just a signing file (this is a UEFI system).

[Gordon Lack (gordon-lack)]

My other system (no UEFI) does seem to be OK on this new kernel - and it is actually running it:

[parent]: cat /proc/version_signature
Ubuntu 4.13.0-38.43+lp1759920.1-generic 4.13.16
[parent]: cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.13.0-38-generic root=UUID=ca819911-be41-4822-8720-4536f278aa56 ro quiet splash enable_mttr_cleanup mtrr_spare_reg_nr=2 mtrr_gran_size=32M mtrr_chunk_size=128M vt.handoff=7
[parent]: dmesg -t | grep -i microcode
microcode: microcode updated early to revision 0x2d, date = 2018-02-07
microcode: sig=0x206a7, pf=0x2, revision=0x2d
microcode: Microcode Update Driver: v2.2.

(ignore the mtrr options - that's to remove some *BAD*gran_size warnings at boot time which mysteriously appeared in Nov 2016 - but I only found out a few days go.)

[Gordon Lack (gordon-lack)]

Well, I *think* I can answer my own question about the signed image.

The signed image was *not* generated by my installing the lp1759920 kernel.
In fact it is the identical file to the one from:
  https://launchpad.net/ubuntu/+archive/primary/+files/kernel-signed-image-4.13.0-38-generic-di_4.13.0-38.43_amd64.udeb
(which is a link from https://launchpad.net/ubuntu/+source/linux-signed).

Given that no similar package is installed on my system for *any* signed kernel image I'm at a loss to know how and when they *do* get signed.

But this does explain why my laptop still hangs - it's because it isn't actually running the lp1759920 kernel.

Is there any (simple-ish) way to generate a signed image from the unsigned one?

[QkiZ (qkiz)]

Works fine.
 ~  cat /proc/version_signature
Ubuntu 4.13.0-38.43+lp1759920.1-lowlatency 4.13.16
 ~  cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.13.0-38-lowlatency root=/dev/mapper/system-root ro rootflags=subvol=@ splash cryptopts=target=pandora-sys,source=/dev/disk/by-uuid/873980ab-cb18-419b-9614-54f23972e042,lvm=system elevator=deadline acpi_rev_override=1 vt.handoff=7
 ~  dmesg -t | grep -i microcode
microcode: microcode updated early to revision 0x24, date = 2018-01-21
microcode: sig=0x306c3, pf=0x20, revision=0x24
microcode: Microcode Update Driver: v2.2.

[Tyler Hicks (tyhicks)]

Any affected 16.04 users that are using the 4.4 kernel can find a test build with the fix here:

https://people.canonical.com/~tyhicks/lp1759920/xenial/

When you provide testing feedback, please remember to include the output from these commands in your comment:

$ cat /proc/version_signature
$ cat /proc/cmdline
$ dmesg -t | grep -i microcode

[John Bryant (bryantj247)]

I've been using the test kernel for 17.10 for a few hours, and it works great!

Ubuntu 4.13.0-38.43+lp1759920.1-generic 4.13.16

BOOT_IMAGE=/boot/vmlinuz-4.13.0-38-generic root=UUID=dd134dc1-6642-41d8-acd1-abc83ab24099 ro quiet splash vt.handoff=7

microcode: microcode updated early to revision 0x2d, date = 2018-02-07
microcode: sig=0x206a7, pf=0x2, revision=0x2d
microcode: Microcode Update Driver: v2.2.

[Kyle Etsler (ketsler)]

@tyhicks

I can confirm that adding 'apparmor=0' to the kernal boot parameters resolved the issue on an effected system.

This was tested on 4.13.0-37-generic, with intel microcode 3.20180312.0

I will proceed with testing kernel: need linux-image-4.13.0-38-generic_4.13.0-38.43+lp1759920.1

[Kyle Etsler (ketsler)]

@tyhicks

I can also confirm that the issue appears to be resolved with linux-image-4.13.0-38-generic_4.13.0-38.43+lp1759920.1

Below is the requested information.

$ uname -a
Linux mercury 4.13.0-38-generic #43+lp1759920.1 SMP Tue Apr 3 22:59:23 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ cat /proc/version_signature
Ubuntu 4.13.0-38.43+lp1759920.1-generic 4.13.16

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.13.0-38-generic root=UUID=5dd1c110-663e-4cf0-9a63-45815590a186 ro quiet splash vt.handoff=7

$ dmesg -t | grep -i microcode
microcode: microcode updated early to revision 0x84, date = 2018-01-21
microcode: sig=0x906e9, pf=0x20, revision=0x84
microcode: Microcode Update Driver: v2.2.

[Aalok Sathe (aalok-sathe)]

This bug has been also reported elsewhere. See here, for instance: https://bugs.launchpad.net/ubuntu/+source/xorg/+bug/1761121

Marked the other link as duplicate of this.

[Hannu Koivisto (azure)]

I can also confirm that the problem doesn't occur with linux-image-4.13.0-38-generic_4.13.0-38.43+lp1759920.1_amd64.deb

cat /proc/version_signature =>
Ubuntu 4.13.0-38.43+lp1759920.1-generic 4.13.16

cat /proc/cmdline =>
BOOT_IMAGE=/vmlinuz-4.13.0-38-generic root=/dev/mapper/kubuntu--vg-root ro quiet splash vt.handoff=7

dmesg -t | grep -i microcode =>
microcode: microcode updated early to revision 0x1f, date = 2018-02-07
microcode: sig=0x306a9, pf=0x10, revision=0x1f
microcode: Microcode Update Driver: v2.2.

[Maxim Loparev (laplandersan)]

Thanks @tyhicks.

$ dmesg | grep microcode
[ 0.000000] microcode: microcode updated early to revision 0x1f, date = 2018-02-07
[ 0.843339] microcode: sig=0x306a9, pf=0x2, revision=0x1f
[ 0.843412] microcode: Microcode Update Driver: v2.2.
$ cat /proc/version_signature
Ubuntu 4.13.0-38.43+lp1759920.1-generic 4.13.16
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.13.0-38-generic root=/dev/mapper/lmvg-ubunturoot ro zswap.enabled=0 quiet

[Rob Johnson (rob.johnson)]

I've tested Ty's 4.4 kernel from comment #78 and can confirm it has let me log in with an LDAP user.
Uname:
  Linux M3520 4.4.0-119-generic #143+lp1759920.1 SMP Wed Apr 4 21:32:55 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

version_signature:
  Ubuntu 4.4.0-119.143+lp1759920.1-generic 4.4.114

cmdline:
  BOOT_IMAGE=/vmlinuz-4.4.0-119-generic root=/dev/mapper/ubuntu--vg-root ro

dmesg:
microcode: CPU0 sig=0x906e9, pf=0x20, revision=0x84
microcode: CPU1 sig=0x906e9, pf=0x20, revision=0x84
microcode: CPU2 sig=0x906e9, pf=0x20, revision=0x84
microcode: CPU3 sig=0x906e9, pf=0x20, revision=0x84
microcode: CPU4 sig=0x906e9, pf=0x20, revision=0x84
microcode: CPU5 sig=0x906e9, pf=0x20, revision=0x84
microcode: CPU6 sig=0x906e9, pf=0x20, revision=0x84
microcode: CPU7 sig=0x906e9, pf=0x20, revision=0x84
microcode: Microcode Update Driver: v2.01 <email address hidden>, Peter Oruba

[QkiZ (qkiz)]

@Tyler when we should expect this fix in official Ubuntu repository?

[Damian Kidd (demomanca)]

Can confirm test kernal fixed my hard lock issue as well:
/proc/version_signature
Ubuntu 4.13.0-38.43+lp1759920.1-generic 4.13.16
/proc/cmdline
BOOT_IMAGE=/@/boot/vmlinuz-4.13.0-38-generic root=UUID=7dcb6f6f-1069-4315-a421-0d60fffecef9 ro rootflags=subvol=@
dmesg -t | grep -i microcode
microcode: microcode updated early to revision 0x2d, date = 2018-02-07
microcode: sig=0x206a7, pf=0x2, revision=0x2d
microcode: Microcode Update Driver: v2.2.

[Tyler Hicks (tyhicks)]

On 04/05/2018 03:25 PM, QkiZ wrote:
> @Tyler when we should expect this fix in official Ubuntu repository?
A little over 2 weeks from now. The week of April 23rd is the current
target.

[Jason P. Thomas (jthomasp)]

Is there any issue with removing the intel-microcode package with the -38 kernel? The intel-microcode package does not seem to get installed on a fresh 16.04.4 system and the Spectre/Meltdown checker referenced earlier shows no vulnerability as the mitigation is all done in the kernel.

[Paul Ashbrook (ashbrook)]

Tyler's amended kernel got me out of a hole today, thanks for the work. I've told my users not to update kernel until this is in the Ubuntu repos.

One observation from me is that VirtualBox 5.2.8 causes my HP Z230 (running Ubuntu 16.04.4) to crash and reboot when I try to start a VM. Not a showstopper for me, and I need to dig around for some clues, but wondered if anyone else had this problem.

[Michael Mauch (michael-mauch)]

Thank you, Tyler, your 4.4 kernel has been working great for half a day, even with the DVB-related patches for my DVB card.

viscacha% cat /proc/version_signature
Ubuntu 4.4.0-119.143+lp1759920.1-generic 4.4.114
viscacha% cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.4.0-119-generic root=UUID=009b027c-db85-4cd1-909a-a658aaab5394 ro nodmraid enable_mtrr_cleanup quiet splash ignore_loglevel i8042.noaux=1 video=vesafb:off intel_idle.max_cstate=1 console=ttyS0,115200 vt.handoff=7
viscacha% dmesg -t | grep -i microcode
microcode: CPU0 microcode updated early to revision 0x2d, date = 2018-02-07
microcode: CPU1 microcode updated early to revision 0x2d, date = 2018-02-07
microcode: CPU2 microcode updated early to revision 0x2d, date = 2018-02-07
microcode: CPU3 microcode updated early to revision 0x2d, date = 2018-02-07
microcode: CPU0 sig=0x206a7, pf=0x2, revision=0x2d
microcode: CPU1 sig=0x206a7, pf=0x2, revision=0x2d
microcode: CPU2 sig=0x206a7, pf=0x2, revision=0x2d
microcode: CPU3 sig=0x206a7, pf=0x2, revision=0x2d
microcode: CPU4 sig=0x206a7, pf=0x2, revision=0x2d
microcode: CPU5 sig=0x206a7, pf=0x2, revision=0x2d
microcode: CPU6 sig=0x206a7, pf=0x2, revision=0x2d
microcode: CPU7 sig=0x206a7, pf=0x2, revision=0x2d
microcode: Microcode Update Driver: v2.01 <email address hidden>, Peter Oruba

[Laurent Dinclaux (laurent-m)]

I uninstalled package intel-microcode and can now use my laptop.

[pinus (pinus)]

After purging intel-microcode package the 38 kernel runs for me.

[Kleber Sacilotto de Souza (kleber-souza)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'. If the problem still exists, change the tag 'verification-needed-trusty' to 'verification-failed-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Erich E. Hoover (ehoover)]

@kleber-souza I don't think that I have anything running trusty that I can test with, but I can easily test on xenial....

[Damian Kidd (demomanca)]

HWE Kernel in xenial-testing resolves the issue as well

/proc/version_signature
Ubuntu 4.13.0-39.44~16.04.1-generic 4.13.16
/proc/cmdline
BOOT_IMAGE=/@/boot/vmlinuz-4.13.0-39-generic root=UUID=7dcb6f6f-1069-4315-a421-0d60fffecef9 ro rootflags=subvol=@
dmesg -t | grep -i microcode
microcode: microcode updated early to revision 0x2d, date = 2018-02-07
microcode: sig=0x206a7, pf=0x2, revision=0x2d
microcode: Microcode Update Driver: v2.2.

[John Bryant (bryantj247)]

I wasn't able to reproduce the problem on Trusty, even with sssd installed.

/proc/version_signature
Ubuntu 4.4.0-119.143~14.04.1-generic 4.4.114

/proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.4.0-119-generic root=UUID=e57a614e-ad1a-449e-954d-ce832d128920 ro quiet splash vt.handoff=7

dmesg -t | grep -i microcode (with CPUs other than 0 removed)
microcode: CPU0 microcode updated early to revision 0x2d, date = 2018-02-07
microcode: CPU0 sig=0x206a7, pf=0x2, revision=0x2d
microcode: Microcode Update Driver: v2.01 <email address hidden>, Peter Oruba

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-artful' to 'verification-done-artful'. If the problem still exists, change the tag 'verification-needed-artful' to 'verification-failed-artful'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Janne Snabb (janne-snabb)]

I tested the new kernel in artful-proposed on a system which failed with the newest kernel + intel-microcode in normal artful repository.

Thus I confirm that the kernel in artful-proposed fixes this issue for me.

$ apt policy linux-image-4.13.0-39-generic
linux-image-4.13.0-39-generic:
  Installed: 4.13.0-39.44
  Candidate: 4.13.0-39.44
  Version table:
 *** 4.13.0-39.44 400
        400 http://archive.ubuntu.com/ubuntu artful-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

$ apt policy intel-microcode
intel-microcode:
  Installed: 3.20180312.0~ubuntu17.10.1
  Candidate: 3.20180312.0~ubuntu17.10.1
  Version table:
 *** 3.20180312.0~ubuntu17.10.1 500
        500 http://fi.archive.ubuntu.com/ubuntu artful-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu artful-security/main amd64 Packages
        100 /var/lib/dpkg/status
     3.20170707.1 500
        500 http://fi.archive.ubuntu.com/ubuntu artful/restricted amd64 Packages

[pdf (pdffs)]

Tested kernel from xenial-proposed on an affected system, and the issue there appears to be resolved.

$ uname -r
4.4.0-120-generic

$ apt policy linux-image-generic
linux-image-generic:
  Installed: 4.4.0.120.126
  Candidate: 4.4.0.120.126
  Version table:
 *** 4.4.0.120.126 400
        400 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     4.4.0.119.125 500
        500 http://mirror.internode.on.net/pub/ubuntu/ubuntu xenial-security/main amd64 Packages
        500 http://mirror.internode.on.net/pub/ubuntu/ubuntu xenial-updates/main amd64 Packages
     4.4.0.21.22 500
        500 http://mirror.internode.on.net/pub/ubuntu/ubuntu xenial/main amd64 Packages

$ apt policy intel-microcode
intel-microcode:
  Installed: 3.20180312.0~ubuntu16.04.1
  Candidate: 3.20180312.0~ubuntu16.04.1
  Version table:
 *** 3.20180312.0~ubuntu16.04.1 500
        500 http://mirror.internode.on.net/pub/ubuntu/ubuntu xenial-security/main amd64 Packages
        500 http://mirror.internode.on.net/pub/ubuntu/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     3.20151106.1 500
        500 http://mirror.internode.on.net/pub/ubuntu/ubuntu xenial/restricted amd64 Packages

[Gordon Lack (gordon-lack)]

I've tested the artful kernel on a UEFI system (and tracked own how the signed kernel is produced from the unsigned on in the debs at he same time - see #76).

It fixes the issue (tag already changed).

[Rob Johnson (rob.johnson)]

@brad-figg #98:
I've tested the Xenial-proposed kernel on a machine that was previously broken (even performed a fresh install to this effect). The proposed kernel (4.13.0-39) fixes the problem.

[Janne Snabb (snabb)]

I may have experienced this issue also on bionic (18.04) with the current kernel version as of today (4.15.0-13.14).

I upgraded one affected system with "do-release-upgrade -d". After completing the upgrade and rebooting, the system froze the same way as with the affected 17.10 kernel and microcode combination. After adding "apparmor=0" to the kernel command line the boot was successful. The system has sssd.

I was not able to perform further testing today. I will try to confirm tomorrow.

Or maybe someone else with an affected system is able to try bionic as well? ISOs available at http://releases.ubuntu.com/18.04/ if needed.

[Gordon Lack (gordon-lack)]

> I may have experienced this issue also on bionic (18.04) with the current kernel version as of today (4.15.0-13.14).

I have a system running bionic and it's fine (see #34). Always has been. It's never run anything else (just over 2 weeks old...). It's always had the "new" (3.20180312.0~ubuntu18.04.1) microcode.
It's running nightly auto-updates.

[Tyler Hicks (tyhicks)]

On 04/10/2018 11:43 AM, Janne Snabb wrote:
> I may have experienced this issue also on bionic (18.04) with the
> current kernel version as of today (4.15.0-13.14).

It wouldn't be this same exact bug as the 4.15.0-13.14 kernel doesn't
call __ptrace_may_access() when deciding to make use of IBPB while doing
a task switch.

It makes me wonder if you didn't accidentally reboot into the 17.10
kernel by accident (or maybe due to some post upgrade bug).

Either way, please leave a comment if it happens again. Thanks for
mentioning it here.

[Michael Mulqueen (michael.mulqueen)]

Is a lack of testing for trusty going to prevent a patch for xenial/artful? We don't have any machines on trusty to test it with, but if it's going to block the whole patch then we might be able to wipe an affected machine and install trusty to test it.

[Dimitri Papadopoulos (dimitri-papadopoulos)]

Experiencing freezes on multiple Ubuntu 16.04 LTS desktops of my organization, all using sssd. Freezes started around April 9th, but we usually deploy new packages after a time period of 5 to 7 days. I have experienced freezes with kernels 4.4.0-119, 4.4.40-116 and even 4.4.0-112 on my own desktop. No more freezes after upgrading to kernel 4.4.0-120.

[Tyler Hicks (tyhicks)]

I was never able to reproduce this issue on Ubuntu 14.04 but, after reviewing the Spectre V2 mitigations in the 3.13 based 14.04 kernel, I think the 14.04 kernel could be affected by this bug. Additionally, I didn't want the decision on when to make use of IBPB to be different in 14.04 than in all of the newer Ubuntu releases (16.04, 17.10, and soon to be 18.04). That's why I marked this bug as affecting 14.04 and submitted a fix for that kernel.

That being said, I don't think that we have any 14.04 users watching this bug so I did some SRU verification work myself. I installed sssd, the 3.13.0-145.194 kernel from trusty-proposed, and experienced no issues. I verified that the /proc/sys/kernel/ibpb_enabled file reported '1', indicating that IBPB was enabled. I also verified that the IBRS/IBPB/retpoline/spectre messages emitted from the kernel at boot time were as-expected. Retpoline was present, IBRS was disabled (due to retpoline), and IBPB was enabled.

[berend (berenddeboer)]

I can confirm that kernel in artful-proposed fixed the issue for me.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.13.0-145.194

---------------
linux (3.13.0-145.194) trusty; urgency=medium

  * linux: 3.13.0-145.194 -proposed tracker (LP: #1761430)

  * intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-
    image-4.13.0-37-generic) (LP: #1759920) // CVE-2017-5715 (Spectre v2 Intel)
    - Revert "UBUNTU: SAUCE: x86/mm: Only set IBPB when the new thread cannot
      ptrace current thread"
    - x86/speculation: Use Indirect Branch Prediction Barrier in context switch

  * DKMS driver builds fail with: Cannot use CONFIG_STACK_VALIDATION=y, please
    install libelf-dev, libelf-devel or elfutils-libelf-devel (LP: #1760876)
    - [Packaging] include the retpoline extractor in the headers

  * retpoline hints: primary infrastructure and initial hints (LP: #1758856)
    - [Packaging] retpoline-extract: flag *0xNNN(%reg) branches
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool on 32bit
    - x86/paravirt, objtool: Annotate indirect calls
    - x86/asm: Stop depending on ptrace.h in alternative.h
    - [Packaging] retpoline -- add safe usage hint support
    - [Packaging] retpoline-check -- only report additions
    - [Packaging] retpoline -- widen indirect call/jmp detection
    - [Packaging] retpoline -- elide %rip relative indirections
    - [Packaging] retpoline -- clear hint information from packages
    - SAUCE: modpost: add discard to non-allocatable whitelist
    - KVM: x86: Make indirect calls in emulator speculation safe
    - KVM: VMX: Make indirect call speculation safe
    - x86/boot, objtool: Annotate indirect jump in secondary_startup_64()
    - SAUCE: early/late -- annotate indirect calls in early/late initialisation
      code
    - SAUCE: vga_set_mode -- avoid jump tables
    - [Config] retpoline -- switch to new format
    - [Packaging] retpoline hints -- handle missing files when RETPOLINE not
      enabled
    - [Packaging] final-checks -- remove check for empty retpoline files

  * retpoline: ignore %cs:0xNNN constant indirections (LP: #1752655)
    - [Packaging] retpoline -- elide %cs:0xNNNN constants on i386

  * Boot crash with Trusty 3.13 (LP: #1757193)
    - Revert "UBUNTU: SAUCE: x86, extable: fix uaccess fixup detection"
    - x86/mm: Expand the exception table logic to allow new handling options

  * Segmentation fault in ldt_gdt_64 (LP: #1755817) // CVE-2017-5754
    - x86/kvm: Rename VMX's segment access rights defines
    - x86/signal/64: Fix SS if needed when delivering a 64-bit signal

 -- Kleber Sacilotto de Souza <email address hidden> Thu, 05 Apr 2018 16:26:39 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-121.145

---------------
linux (4.4.0-121.145) xenial; urgency=medium

  * linux: 4.4.0-121.145 -proposed tracker (LP: #1763687)

  * Ubuntu-4.4.0-120.144 fails to boot on arm64* hardware (LP: #1763644)
    - [Config] arm64: disable BPF_JIT_ALWAYS_ON

linux (4.4.0-120.144) xenial; urgency=medium

  * linux: 4.4.0-120.144 -proposed tracker (LP: #1761438)

  * intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-
    image-4.13.0-37-generic) (LP: #1759920) // CVE-2017-5715 (Spectre v2 Intel)
    - Revert "x86/mm: Only set IBPB when the new thread cannot ptrace current
      thread"
    - x86/speculation: Use Indirect Branch Prediction Barrier in context switch

  * DKMS driver builds fail with: Cannot use CONFIG_STACK_VALIDATION=y, please
    install libelf-dev, libelf-devel or elfutils-libelf-devel (LP: #1760876)
    - [Packaging] include the retpoline extractor in the headers

  * retpoline hints: primary infrastructure and initial hints (LP: #1758856)
    - [Packaging] retpoline-extract: flag *0xNNN(%reg) branches
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool on 32bit
    - x86/paravirt, objtool: Annotate indirect calls
    - x86/asm: Stop depending on ptrace.h in alternative.h
    - [Packaging] retpoline -- add safe usage hint support
    - [Packaging] retpoline-check -- only report additions
    - [Packaging] retpoline -- widen indirect call/jmp detection
    - [Packaging] retpoline -- elide %rip relative indirections
    - [Packaging] retpoline -- clear hint information from packages
    - SAUCE: modpost: add discard to non-allocatable whitelist
    - KVM: x86: Make indirect calls in emulator speculation safe
    - KVM: VMX: Make indirect call speculation safe
    - x86/boot, objtool: Annotate indirect jump in secondary_startup_64()
    - SAUCE: early/late -- annotate indirect calls in early/late initialisation
      code
    - SAUCE: vga_set_mode -- avoid jump tables
    - [Config] retpoline -- switch to new format
    - [Packaging] final-checks -- remove check for empty retpoline files

  * Xenial update to 4.4.117 stable release (LP: #1756860)
    - IB/mlx4: Fix incorrectly releasing steerable UD QPs when have only ETH ports
    - PM / devfreq: Propagate error from devfreq_add_device()
    - s390: fix handling of -1 in set{,fs}[gu]id16 syscalls
    - ARM: dts: STi: Add gpio polarity for "hdmi,hpd-gpio" property
    - arm: spear600: Add missing interrupt-parent of rtc
    - arm: spear13xx: Fix dmas cells
    - arm: spear13xx: Fix spics gpio controller's warning
    - ALSA: seq: Fix regression by incorrect ioctl_mutex usages
    - KVM/x86: Reduce retpoline performance impact in slot_handle_level_range(),
      by always inlining iterator helper methods
    - x86/cpu: Change type of x86_cache_size variable to unsigned int
    - drm/radeon: adjust tested variable
    - rtc-opal: Fix handling of firmware error codes, prevent busy loops
    - ext4: save error to disk in __ext4_grp_locked_error()
    - ext4: correct documentation for grpid mount option
    - mm: hide a #warning fo...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.13.0-39.44

---------------
linux (4.13.0-39.44) artful; urgency=medium

  * linux: 4.13.0-39.44 -proposed tracker (LP: #1761456)

  * intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-
    image-4.13.0-37-generic) (LP: #1759920) // CVE-2017-5715 (Spectre v2
    Intel) // CVE-2017-5754
    - x86/mm: Reinitialize TLB state on hotplug and resume

  * intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-
    image-4.13.0-37-generic) (LP: #1759920) // CVE-2017-5715 (Spectre v2 Intel)
    - Revert "x86/mm: Only set IBPB when the new thread cannot ptrace current
      thread"
    - x86/speculation: Use Indirect Branch Prediction Barrier in context switch

  * DKMS driver builds fail with: Cannot use CONFIG_STACK_VALIDATION=y, please
    install libelf-dev, libelf-devel or elfutils-libelf-devel (LP: #1760876)
    - [Packaging] include the retpoline extractor in the headers

  * retpoline hints: primary infrastructure and initial hints (LP: #1758856)
    - [Packaging] retpoline-extract: flag *0xNNN(%reg) branches
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool on 32bit
    - x86/paravirt, objtool: Annotate indirect calls
    - [Packaging] retpoline -- add safe usage hint support
    - [Packaging] retpoline-check -- only report additions
    - [Packaging] retpoline -- widen indirect call/jmp detection
    - [Packaging] retpoline -- elide %rip relative indirections
    - [Packaging] retpoline -- clear hint information from packages
    - KVM: x86: Make indirect calls in emulator speculation safe
    - KVM: VMX: Make indirect call speculation safe
    - x86/boot, objtool: Annotate indirect jump in secondary_startup_64()
    - SAUCE: early/late -- annotate indirect calls in early/late initialisation
      code
    - SAUCE: vga_set_mode -- avoid jump tables
    - [Config] retpoline -- switch to new format
    - [Packaging] retpoline hints -- handle missing files when RETPOLINE not
      enabled
    - [Packaging] final-checks -- remove check for empty retpoline files

  * retpoline: ignore %cs:0xNNN constant indirections (LP: #1752655)
    - [Packaging] retpoline -- elide %cs:0xNNNN constants on i386

  * zfs system process hung on container stop/delete (LP: #1754584)
    - SAUCE: Fix non-prefaulted page deadlock (LP: #1754584)

  * zfs-linux 0.6.5.11-1ubuntu5 ADT test failure with linux 4.15.0-1.2
    (LP: #1737761)
    - SAUCE: (noup) Update zfs to 0.6.5.11-1ubuntu3.2

  * AT_BASE_PLATFORM in AUXV is absent on kernels available on Ubuntu 17.10
    (LP: #1759312)
    - powerpc/64s: Fix NULL AT_BASE_PLATFORM when using DT CPU features

  * btrfs and tar sparse truncate archives (LP: #1757565)
    - Btrfs: move definition of the function btrfs_find_new_delalloc_bytes
    - Btrfs: fix reported number of inode blocks after buffered append writes

  * efifb broken on ThunderX-based Gigabyte nodes (LP: #1758375)
    - drivers/fbdev/efifb: Allow BAR to be moved instead of claiming it

  * Intel i40e PF reset due to incorrect MDD detection (continues...)
    (LP: #1723127)
    - i40e/i40ev...

[bmaupin (bmaupin)]

This is *not* fixed for Xenial. I had originally applied the workaround in the bug description to get my computer to boot again. After seeing a "fixed" status for this bug, I updated intel-microcode. After I rebooted, my computer would not boot to the kernel which had the microcode applied (4.4.0-122). It would only boot to the previous kernel without the microcode (4.4.0-119).

$ tail -n 5 /var/log/apt/history.log
Start-Date: 2018-05-05 06:06:43
Commandline: apt install intel-microcode
Upgrade: intel-microcode:amd64 (3.20180108.0+really20170707ubuntu16.04.1, 3.20180312.0~ubuntu16.04.1)
End-Date: 2018-05-05 06:06:57

$ uname -a
Linux thinkpad 4.4.0-119-generic #143-Ubuntu SMP Mon Apr 2 16:08:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ dpkg -l | grep linux-image-4 | grep ^i
ii linux-image-4.4.0-112-generic 4.4.0-112.135 amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii linux-image-4.4.0-119-generic 4.4.0-119.143 amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii linux-image-4.4.0-122-generic 4.4.0-122.146 amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial

$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 58
model name : Intel(R) Core(TM) i5-3317U CPU @ 1.70GHz
stepping : 9
microcode : 0x1c
cpu MHz : 893.761
cache size : 3072 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm epb retpoline kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 3391.94
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

[bmaupin (bmaupin)]

Since this bug was marked as fixed I'm not sure what the best way to proceed is. I went ahead and filed a new bug: https://bugs.launchpad.net/ubuntu/+source/intel-microcode/+bug/1769335

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.2.0-150.197

---------------
linux (3.2.0-150.197) precise; urgency=medium

  * precise/linux: 3.2.0-150.197 -proposed tracker (LP: #1919172)

  * CVE-2021-27365
    - scsi: iscsi: Verify lengths on passthrough PDUs
    - sysfs: Add sysfs_emit and sysfs_emit_at to format sysfs output
    - scsi: iscsi: Ensure sysfs attributes are limited to PAGE_SIZE

  * CVE-2021-27363 // CVE-2021-27364
    - scsi: iscsi: Restrict sessions and handles to admin capabilities

  * CVE-2021-27364
    - scsi: iscsi: respond to netlink with unicast when appropriate
    - Add file_ns_capable() helper function for open-time capability checking
    - net: Add variants of capable for use on on sockets
    - netlink: Make the sending netlink socket availabe in NETLINK_CB

 -- Thadeu Lima de Souza Cascardo <email address hidden> Mon, 05 Apr 2021 14:23:29 -0300