intel-microcode not included 20.04 initrd

[Chris Iverach-Brereton]

We have several factory-fresh Mitac industrial motherboards (https://www.mitacmct.com/IndustrialMotherboard_PH13FEI_PH13FEI) that we are unable to install Ubuntu Server 18.04.5 on. When we boot from the installation media we get the following error and the installation hangs (see attached screenshot, I've transcribed the error below too)

```
[0.000000] [Firmware Bug]: TSC_DEADLINE disabled due to Errata; please update microcode to version: 0x52 (or later)
[0.157025] ACPI Exception: AE_BAD_PARAMETER, Could not install PciConfig handler for Root Bridge PCI0 (20170831/evrgnini-241)
```

Googling the microcode bug, the normal advice appears to be "boot into an older, functioning kernel, update the `intel-microcode` package, and then reboot." But in this situation there is no older kernel to boot into; there is no OS installed _at all_ on these boards yet. I'm booting from a USB drive loaded with the latest version of Ubuntu Server 18.04.

The only work-around I've found that works so far is to install Ubuntu Desktop 18.04, which can boot successfully on these boards, perform the minimal desktop installation, then use the desktop install to update the `intel-microcode` package, reboot, and then finally install Ubuntu Server. This is obviously not convenient needing to install a complete desktop OS just so I can update the microcode, and then immediately wiping it to install what I wanted in the first place.

I have not tried Ubuntu 20.x or 21.x; I need to use Ubuntu 18.04 because these motherboards are being installed into robots that will be running ROS Melodic (ROS Noetic runs on 20.04, but these robots don't have Noetic support yet).

I have tried the following to modify the Ubuntu Server ISO, without success:

1- Adding the microcode to initrd
---------------------------------------

I unpacked the contents of the Ubuntu Server ISO, decompressed install/initrd.gz, and used the bash script here https://www.kernel.org/doc/html/latest/x86/microcode.html to add the intel-microcode firmware to the image. Then I recompressed initrd.gz, and rebuild the ISO using xorisso. This did not have any effect: Ubuntu Server still encountered the same error when trying to install.

2- Using initrd from Ubuntu Desktop 18.04.5
----------------------------------------------

I downloaded & unpacked Ubuntu Desktop 18.04.5 and replaced Ubuntu Server's install/initrd.gz with Ubuntu Desktop's casper/initrd (updating txt.cfg and grub.cfg to load the uncompressed initrd instead of the .gz that Ubuntu Server normally uses). Then I rebuilt the Ubuntu Server with xorisso again.

The resulted in the system booting, but immediately dropping into an initramfs busybox shell, saying boot files were missing. I assume this is because Ubuntu Server uses the `boot=casper` flag in order to boot into the live desktop session.

At this point I'm running out of ideas for how to be able to make Ubuntu Server's installation media work with these boards. Ubuntu Desktop has no problem booting on them, so it's obviously something specific to the Ubuntu Server image, but I'm not sure exactly what or how to fix it.

If you need any additional information/logs/etc... please let me know and I'll be happy to provide them.

Thanks!

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1938588/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Chris Iverach-Brereton (civerachb)]

If I boot off the ISO in a VM and use alt+Left I can get to a terminal. `uname -a` reports:

```
Linux (none) 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 GNU/Linux
```

I couldn't find that specific kernel package with the search tool, so please feel free to change the package as necessary to something more specific than just "linux"

[Brian Murray (brian-murray)]

This is because the intel-microcode package is not included on the server iso for Ubuntu 18.04 but it is for the desktop iso. This also seems to be the case for Ubuntu 20.04.

[Chris Iverach-Brereton (civerachb)]

Are there plans to modify the server ISO for 18.04 and 20.04 to address this limitation? Or will I need to modify the initrd file myself and create a customized image for these boards?

[Brian Murray (brian-murray)]

We are looking at updating the server iso for the 20.04.3 point release which is coming up in a couple of weeks. However, there are no further point releases of 18.04 (and subsequently new isos) scheduled so this is unlikely to be fixed for that release.

[Chris Iverach-Brereton (civerachb)]

That's disappointing, but understandable. Is there any chance you could provide me with a link to a guide to explain how I could edit the ISO on my own to make a custom ISO for these boards? I've already got some customized preseed files anyway, so making further modifications to the base Ubuntu Server image isn't a big deal. I just don't know what the exact process for adding the microcode would be.

My previous efforts using the example script from https://www.kernel.org/doc/html/latest/x86/microcode.html to add the microcode to initrd weren't successful. But I'm not even sure if that's the correct process, or if there's an easier/better way to do it.

Thanks!

[Dimitri John Ledkov (xnox)]

inclusion of the package on the iso is one thing; it is a different thing to build the boot initrd with microcode included. I do not believe that d-i base installation media ever create d-i initrd with microcode included. Thus even if package is included on the ISO it will not be present in the d-i initrd.

Live based isos all should have microcode in the initrd they boot.

Given that the boards are buggy, it's best to download the firmware update from your motherboard manufacturer to update the firmware & cpu microcode of the board itself using like Freedos boot. Then from that point on, all/any boots and installs of anything will at least have working microcode.

[Chris Iverach-Brereton (civerachb)]

Yeah, I knew I'd need to rebuild the initrd to include the microcode vs just including it in the ISO. It would be really nice if Ubuntu's non-live ISOs included the microcode in initrd like the Live ISOs do though.

I'm trying to get a hold of someone from the manufacturer and see if they can update their firmware to address the problem, but no luck so far, unfortunately.

In the meantime I'll give FreeDOS a shot and see if that works. I already know that doing a full Ubuntu Desktop installation fixes it too, but that obviously takes a long time. If FreeDOS works we can probably use that as a work-around for now until (hopefully) the manufacturer addresses their board problem.