Dell 5280 (id=0x00050654/06-55-04) hangs on warm reboot after upgrading intel-microcode package

[mpikyb]

We have brand-new Dell T 5280 workstations (Intel Xeon W-2133 3.6 GHz (3.9 GHz Turbo), 6 Cores) with latest BIOS (1.13.1) and Ubuntu 18.04.3 LTS

$ uname -a
Linux hostname 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -rd
Description: Ubuntu 18.04.3 LTS
Release: 18.04

$ dpkg -l | grep linux-image-4 | grep ^i
ii linux-image-4.15.0-66-generic 4.15.0-66.75 amd64 Signed kernel image generic
ii linux-image-4.15.0-70-generic 4.15.0-70.79 amd64 Signed kernel image generic

$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) W-2133 CPU @ 3.60GHz
stepping : 4
microcode : 0x200005e
cpu MHz : 1200.410
cache size : 8448 KB
physical id : 0
siblings : 12
[...]

After upgrading the 'intel-microcode' package from 3.20180312.0~ubuntu18.04.1 to 3.20191115.1ubuntu0.18.04.1 the machine hangs upon a warm restart (reboot --> black screen, cursor blinking in the top left corner). Powering off the machine and turning it on again will boot just fine.

Removing the intel-microcode package will and rebooting twice will fix the issue. After reinstalling version 3.20180312.0~ubuntu18.04.1 the machine will still be booting correctly.

[Steve Beattie (sbeattie)]

Hi,

Sorry for the difficulty you are having.

I think I've done the conversion correctly, but can you please confirm that the output of iucode-tool -S on these systems gives the processor signature as '0x00050654'?

If that's the case, I believe that this is https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/issues/21 which is still waiting on Intel diagnose. That said, it's probably worth us reverting to the 0x2000064 version of that specific microcode.

The specific microcode your bios currently has in place, 0x200005e, is also available from the https://launchpad.net/ubuntu/+source/intel-microcode/3.20190514.0ubuntu0.19.04.3 version of the package; if needed (by you or others), the binary package can be directly downloaded from that page, even though it is no longer present in the archive.

[mpikyb (mpikyb)]

Thanks for looking into this!

$ iucode-tool -S
iucode-tool: system has processor(s) with signature 0x00050654

So, yes, you did the conversion correctly.
BTW we are using legacy MBR boot mode in case this matters?

[Steve Beattie (sbeattie)]

I don't believe the boot mode should matter.

I have prepared a proposed revert for bionic in the Ubuntu Security Proposed ppa: https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/ that downgrades the microcode from 0x2000065 to 0x2000064 for processors with signature 0x00050654. It would be great if you could confirm the version 3.20191115.1ubuntu0.18.04.2 stops the hanging behavior after warm reboots for your systems.

I'm in the process of preparing packages for the rest of the supported Ubuntu releases.

Thanks again.

[Steve Beattie (sbeattie)]

Packages for all Ubuntu releases are now available for testing in the Ubuntu Security Proposed ppa: https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/ ; as this ppa is for packages we are testing to release as security updates, please do not leave this ppa enabled by default.

Any testing feedback would be appreciated. Thanks!

[mpikyb (mpikyb)]

OK, just tried the upgrade:

$ apt list --upgradable
Listing... Done
intel-microcode/bionic-updates,bionic-security 3.20191115.1ubuntu0.18.04.1 amd64 [upgradable from: 3.20180312.0~ubuntu18.04.1]
linux-headers-generic/bionic-updates 4.15.0.72.74 amd64 [upgradable from: 4.15.0.70.72]
linux-libc-dev/bionic-updates 4.15.0-72.81 amd64 [upgradable from: 4.15.0-70.79]
mpiadmin@chive:~$ sudo apt install --only-upgrade intel-microcode
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  intel-microcode
1 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 2.409 kB of archives.
After this operation, 1.464 kB of additional disk space will be used.
Get:1 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 intel-microcode amd64 3.20191115.1ubuntu0.18.04.1 [2.409 kB]
Fetched 2.409 kB in 0s (10,6 MB/s)
(Reading database ... 241276 files and directories currently installed.)
Preparing to unpack .../intel-microcode_3.20191115.1ubuntu0.18.04.1_amd64.deb ...
Unpacking intel-microcode (3.20191115.1ubuntu0.18.04.1) over (3.20180312.0~ubuntu18.04.1) ...
Setting up intel-microcode (3.20191115.1ubuntu0.18.04.1) ...
update-initramfs: deferring update (trigger activated)
intel-microcode: microcode will be updated at next boot
Processing triggers for initramfs-tools (0.130ubuntu3.9) ...
update-initramfs: Generating /boot/initrd.img-4.15.0-70-generic

$ sudo reboot
(wait for reboot, log in again)
$ sudo reboot
(need to do this twice to make sure new kernel is in place)

--> still hang on warm reboot

[mpikyb (mpikyb)]

Sorry, wrong repository - please ignore the previous post (I was still using the broken package version)

Second attempt:

$ sudo add-apt-repository ppa:ubuntu-security-proposed/ppa
$ sudo apt-get update
$ sudo apt install intel-microcode
$ sudo reboot

$ sudo reboot

$ apt policy intel-microcode
intel-microcode:
  Installed: 3.20191115.1ubuntu0.18.04.2
  Candidate: 3.20191115.1ubuntu0.18.04.2
  Version table:
 *** 3.20191115.1ubuntu0.18.04.2 500
        500 http://ppa.launchpad.net/ubuntu-security-proposed/ppa/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status
     3.20191115.1ubuntu0.18.04.1 500
        500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     3.20180312.0~ubuntu18.04.1 500
        500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

==> the version in the Ubuntu Security Proposed PPA seems to be working fine.

Thanks!

[Steve Beattie (sbeattie)]

Great, thank you for the feedback!

[Launchpad Janitor (janitor)]

This bug was fixed in the package intel-microcode - 3.20191115.1ubuntu0.16.04.2

---------------
intel-microcode (3.20191115.1ubuntu0.16.04.2) xenial-security; urgency=medium

  * REGRESSION UPDATE: warm reboots cause hangs on certain Skylake
    processors (LP: #1854764)
    + Reverted microcode (from revision 0x2000065):
      sig 0x00050654, pf_mask 0xb7, 2019-07-31, rev 0x2000064, size 33792

 -- Steve Beattie <email address hidden> Mon, 02 Dec 2019 12:53:04 -0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package intel-microcode - 3.20191115.1ubuntu0.19.04.2

---------------
intel-microcode (3.20191115.1ubuntu0.19.04.2) disco-security; urgency=medium

  * REGRESSION UPDATE: warm reboots cause hangs on certain Skylake
    processors (LP: #1854764)
    + Reverted microcode (from revision 0x2000065):
      sig 0x00050654, pf_mask 0xb7, 2019-07-31, rev 0x2000064, size 33792

 -- Steve Beattie <email address hidden> Mon, 02 Dec 2019 12:39:25 -0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package intel-microcode - 3.20191115.1ubuntu0.19.10.2

---------------
intel-microcode (3.20191115.1ubuntu0.19.10.2) eoan-security; urgency=medium

  * REGRESSION UPDATE: warm reboots cause hangs on certain Skylake
    processors (LP: #1854764)
    + Reverted microcode (from revision 0x2000065):
      sig 0x00050654, pf_mask 0xb7, 2019-07-31, rev 0x2000064, size 33792

 -- Steve Beattie <email address hidden> Mon, 02 Dec 2019 12:15:00 -0800

[Jonny Spoglemon (dr-spooglemon)]

Just clicked on the microcode update and now my desktop wont start. I have Mint installed on my computer using the Cinnamon desktop.

[Steve Beattie (sbeattie)]

Hey Jonny, sorry you are having difficulties. Can you open a separate bug against the intel-microcode package via https://bugs.launchpad.net/ubuntu/+source/intel-microcode/+filebug so that we can troubleshoot it separately there?

(For diagnosing the issue, passing the 'dis_ucode_ldr' option on the boot command line will prevent the microcode from loading. The output of the 'iucode-tool -S' command would be useful, too.)

[Thomas Ward (teward)]

This landed in Bionic but the changelog syntax was missing the bug number in such a way that would autoclose the Bionic task. Marking it as "Fix Released" for the purposes of doing what the system should've done (but didn't).

[Launchpad Janitor (janitor)]

This bug was fixed in the package intel-microcode - 3.20191115.1ubuntu2

---------------
intel-microcode (3.20191115.1ubuntu2) focal; urgency=medium

  * REGRESSION UPDATE: warm reboots cause hangs on certain Skylake
    processors (LP: #1854764)
    + Reverted microcode (from revision 0x2000065):
      sig 0x00050654, pf_mask 0xb7, 2019-07-31, rev 0x2000064, size 33792

 -- Steve Beattie <email address hidden> Mon, 02 Dec 2019 10:23:56 -0800

[PhilJ (zfil)]

Hello Steve @sbeattie

Just installed intel-microcode 3.20200609.0ubuntu0.18.04.0 on a Dell Precision 7530 using 18.04 and I experience hangs on warm reboots without a power source attached or waking up from sleep.

I've tried BIOS 1.12.1 and 1.13.1 for the same result.

iucode-tool: system has processor(s) with signature 0x000906ea

Previously I was using the intel-microcode package from https://launchpad.net/~vicamo/+archive/ubuntu/ppa-1862751
https://bugs.launchpad.net/ubuntu/+source/intel-microcode/+bug/1858810
and this was fixing the problem.

Cheers

[Stéphane Verdy (sverdy)]

@PhilJ, I think you hit this bug: https://bugs.launchpad.net/bugs/1882890
Could you please try to update the intel-microcode package to version 3.20200609.0ubuntu0.18.04.1 (just uploaded to bionic-security a few hours ago) and confirm it fixes your problem?

[PhilJ (zfil)]

Hello @sverdy

Just did the upgrade and unfortunately this doesn't changes anything ...