Dell 5280 hangs on warm reboot after upgrading intel-microcode package

Bug #1854764 reported by mpikyb on 2019-12-02
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
intel-microcode (Ubuntu)
Status tracked in Focal
Xenial
High
Steve Beattie
Bionic
High
Steve Beattie
Disco
High
Steve Beattie
Eoan
High
Steve Beattie
Focal
High
Steve Beattie

Bug Description

We have brand-new Dell T 5280 workstations (Intel Xeon W-2133 3.6 GHz (3.9 GHz Turbo), 6 Cores) with latest BIOS (1.13.1) and Ubuntu 18.04.3 LTS

$ uname -a
Linux hostname 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -rd
Description: Ubuntu 18.04.3 LTS
Release: 18.04

$ dpkg -l | grep linux-image-4 | grep ^i
ii linux-image-4.15.0-66-generic 4.15.0-66.75 amd64 Signed kernel image generic
ii linux-image-4.15.0-70-generic 4.15.0-70.79 amd64 Signed kernel image generic

$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) W-2133 CPU @ 3.60GHz
stepping : 4
microcode : 0x200005e
cpu MHz : 1200.410
cache size : 8448 KB
physical id : 0
siblings : 12
[...]

After upgrading the 'intel-microcode' package from 3.20180312.0~ubuntu18.04.1 to 3.20191115.1ubuntu0.18.04.1 the machine hangs upon a warm restart (reboot --> black screen, cursor blinking in the top left corner). Powering off the machine and turning it on again will boot just fine.

Removing the intel-microcode package will and rebooting twice will fix the issue. After reinstalling version 3.20180312.0~ubuntu18.04.1 the machine will still be booting correctly.

Steve Beattie (sbeattie) wrote :

Hi,

Sorry for the difficulty you are having.

I think I've done the conversion correctly, but can you please confirm that the output of iucode-tool -S on these systems gives the processor signature as '0x00050654'?

If that's the case, I believe that this is https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/issues/21 which is still waiting on Intel diagnose. That said, it's probably worth us reverting to the 0x2000064 version of that specific microcode.

The specific microcode your bios currently has in place, 0x200005e, is also available from the https://launchpad.net/ubuntu/+source/intel-microcode/3.20190514.0ubuntu0.19.04.3 version of the package; if needed (by you or others), the binary package can be directly downloaded from that page, even though it is no longer present in the archive.

Changed in intel-microcode (Ubuntu):
assignee: nobody → Steve Beattie (sbeattie)
status: New → Confirmed
importance: Undecided → High
mpikyb (mpikyb) wrote :

Thanks for looking into this!

$ iucode-tool -S
iucode-tool: system has processor(s) with signature 0x00050654

So, yes, you did the conversion correctly.
BTW we are using legacy MBR boot mode in case this matters?

Steve Beattie (sbeattie) wrote :

I don't believe the boot mode should matter.

I have prepared a proposed revert for bionic in the Ubuntu Security Proposed ppa: https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/ that downgrades the microcode from 0x2000065 to 0x2000064 for processors with signature 0x00050654. It would be great if you could confirm the version 3.20191115.1ubuntu0.18.04.2 stops the hanging behavior after warm reboots for your systems.

I'm in the process of preparing packages for the rest of the supported Ubuntu releases.

Thanks again.

Steve Beattie (sbeattie) wrote :

Packages for all Ubuntu releases are now available for testing in the Ubuntu Security Proposed ppa: https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/ ; as this ppa is for packages we are testing to release as security updates, please do not leave this ppa enabled by default.

Any testing feedback would be appreciated. Thanks!

Changed in intel-microcode (Ubuntu Eoan):
status: New → In Progress
Changed in intel-microcode (Ubuntu Focal):
status: Confirmed → In Progress
Changed in intel-microcode (Ubuntu Eoan):
importance: Undecided → High
Changed in intel-microcode (Ubuntu Disco):
status: New → In Progress
importance: Undecided → High
Changed in intel-microcode (Ubuntu Bionic):
status: New → In Progress
importance: Undecided → High
Changed in intel-microcode (Ubuntu Xenial):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Steve Beattie (sbeattie)
Changed in intel-microcode (Ubuntu Bionic):
assignee: nobody → Steve Beattie (sbeattie)
Changed in intel-microcode (Ubuntu Disco):
assignee: nobody → Steve Beattie (sbeattie)
Changed in intel-microcode (Ubuntu Eoan):
assignee: nobody → Steve Beattie (sbeattie)
mpikyb (mpikyb) wrote :

OK, just tried the upgrade:

$ apt list --upgradable
Listing... Done
intel-microcode/bionic-updates,bionic-security 3.20191115.1ubuntu0.18.04.1 amd64 [upgradable from: 3.20180312.0~ubuntu18.04.1]
linux-headers-generic/bionic-updates 4.15.0.72.74 amd64 [upgradable from: 4.15.0.70.72]
linux-libc-dev/bionic-updates 4.15.0-72.81 amd64 [upgradable from: 4.15.0-70.79]
mpiadmin@chive:~$ sudo apt install --only-upgrade intel-microcode
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  intel-microcode
1 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 2.409 kB of archives.
After this operation, 1.464 kB of additional disk space will be used.
Get:1 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 intel-microcode amd64 3.20191115.1ubuntu0.18.04.1 [2.409 kB]
Fetched 2.409 kB in 0s (10,6 MB/s)
(Reading database ... 241276 files and directories currently installed.)
Preparing to unpack .../intel-microcode_3.20191115.1ubuntu0.18.04.1_amd64.deb ...
Unpacking intel-microcode (3.20191115.1ubuntu0.18.04.1) over (3.20180312.0~ubuntu18.04.1) ...
Setting up intel-microcode (3.20191115.1ubuntu0.18.04.1) ...
update-initramfs: deferring update (trigger activated)
intel-microcode: microcode will be updated at next boot
Processing triggers for initramfs-tools (0.130ubuntu3.9) ...
update-initramfs: Generating /boot/initrd.img-4.15.0-70-generic

$ sudo reboot
(wait for reboot, log in again)
$ sudo reboot
(need to do this twice to make sure new kernel is in place)

--> still hang on warm reboot

mpikyb (mpikyb) wrote :

Sorry, wrong repository - please ignore the previous post (I was still using the broken package version)

Second attempt:

$ sudo add-apt-repository ppa:ubuntu-security-proposed/ppa
$ sudo apt-get update
$ sudo apt install intel-microcode
$ sudo reboot

$ sudo reboot

$ apt policy intel-microcode
intel-microcode:
  Installed: 3.20191115.1ubuntu0.18.04.2
  Candidate: 3.20191115.1ubuntu0.18.04.2
  Version table:
 *** 3.20191115.1ubuntu0.18.04.2 500
        500 http://ppa.launchpad.net/ubuntu-security-proposed/ppa/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status
     3.20191115.1ubuntu0.18.04.1 500
        500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     3.20180312.0~ubuntu18.04.1 500
        500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

==> the version in the Ubuntu Security Proposed PPA seems to be working fine.

Thanks!

Steve Beattie (sbeattie) wrote :

Great, thank you for the feedback!

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package intel-microcode - 3.20191115.1ubuntu0.16.04.2

---------------
intel-microcode (3.20191115.1ubuntu0.16.04.2) xenial-security; urgency=medium

  * REGRESSION UPDATE: warm reboots cause hangs on certain Skylake
    processors (LP: #1854764)
    + Reverted microcode (from revision 0x2000065):
      sig 0x00050654, pf_mask 0xb7, 2019-07-31, rev 0x2000064, size 33792

 -- Steve Beattie <email address hidden> Mon, 02 Dec 2019 12:53:04 -0800

Changed in intel-microcode (Ubuntu Xenial):
status: In Progress → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package intel-microcode - 3.20191115.1ubuntu0.19.04.2

---------------
intel-microcode (3.20191115.1ubuntu0.19.04.2) disco-security; urgency=medium

  * REGRESSION UPDATE: warm reboots cause hangs on certain Skylake
    processors (LP: #1854764)
    + Reverted microcode (from revision 0x2000065):
      sig 0x00050654, pf_mask 0xb7, 2019-07-31, rev 0x2000064, size 33792

 -- Steve Beattie <email address hidden> Mon, 02 Dec 2019 12:39:25 -0800

Changed in intel-microcode (Ubuntu Disco):
status: In Progress → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package intel-microcode - 3.20191115.1ubuntu0.19.10.2

---------------
intel-microcode (3.20191115.1ubuntu0.19.10.2) eoan-security; urgency=medium

  * REGRESSION UPDATE: warm reboots cause hangs on certain Skylake
    processors (LP: #1854764)
    + Reverted microcode (from revision 0x2000065):
      sig 0x00050654, pf_mask 0xb7, 2019-07-31, rev 0x2000064, size 33792

 -- Steve Beattie <email address hidden> Mon, 02 Dec 2019 12:15:00 -0800

Changed in intel-microcode (Ubuntu Eoan):
status: In Progress → Fix Released

Just clicked on the microcode update and now my desktop wont start. I have Mint installed on my computer using the Cinnamon desktop.

Steve Beattie (sbeattie) wrote :

Hey Jonny, sorry you are having difficulties. Can you open a separate bug against the intel-microcode package via https://bugs.launchpad.net/ubuntu/+source/intel-microcode/+filebug so that we can troubleshoot it separately there?

(For diagnosing the issue, passing the 'dis_ucode_ldr' option on the boot command line will prevent the microcode from loading. The output of the 'iucode-tool -S' command would be useful, too.)

Thomas Ward (teward) wrote :

This landed in Bionic but the changelog syntax was missing the bug number in such a way that would autoclose the Bionic task. Marking it as "Fix Released" for the purposes of doing what the system should've done (but didn't).

Changed in intel-microcode (Ubuntu Bionic):
status: In Progress → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package intel-microcode - 3.20191115.1ubuntu2

---------------
intel-microcode (3.20191115.1ubuntu2) focal; urgency=medium

  * REGRESSION UPDATE: warm reboots cause hangs on certain Skylake
    processors (LP: #1854764)
    + Reverted microcode (from revision 0x2000065):
      sig 0x00050654, pf_mask 0xb7, 2019-07-31, rev 0x2000064, size 33792

 -- Steve Beattie <email address hidden> Mon, 02 Dec 2019 10:23:56 -0800

Changed in intel-microcode (Ubuntu Focal):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.