FIPS/OEM installation compatibility is unclear to the end-user

Bug #1978890 reported by Kyler Hornor
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
subiquity (Ubuntu)
Fix Committed
Undecided
Dan Bungert
ubiquity (Ubuntu)
New
Undecided
Unassigned
ubuntu-advantage-tools (Ubuntu)
New
Undecided
Unassigned
ubuntu-drivers-common (Ubuntu)
New
Undecided
Unassigned
update-manager (Ubuntu)
New
Undecided
Unassigned

Bug Description

[Overall Summary]

Converting to cover all oem/fips compatibility issues with ua/installers/update-manager. These projects are mostly silo'd, so when they all converge it creates a confusing and frustrating experience for the user.

At it's core, the problem is that both fips and oem us GRUB_FLAVOUR_ORDER to select the preferred kernel to boot from, disregarding versioning.

The main issues are:
1. ubuntu-drivers should not attempt to `oem-ify` a `fipsified` machine
2. ua tool should not attempt to `fipsify` an oem machine
3. subiquity should mention that drivers page is potentially making machine realtime & fips incompatible

Below are some reproducible examples of issues:

---
(Subiquity installer case)
[Summary]
A recent change to the subiquity snap adds support for installing oem drivers at time of instance install. If the user installs these packages, then attempts to install the fips packages post-install, fips will install as expected, but the system will always boot to the oem kernel.

[Expected Behavior]
Messaging should clearly indicate that installing the oem packages will make the environment incompatible with fips/RT kernel/ etc.

[Observed Behavior]
Subiquity just offers additional drivers, without clarifying the compatibility complications.

[Replication Steps]
(Using Dell Inc. Precision 7920 Tower/060K5C)
1. Install from current focal ISO
2. Confirm driver installation on the oem gui page
3. Install ua client/fips
4. Reboot
5. Observe kernel version (oem)

---
(update-manager case)
[Summary]
A feature was added to allow for post-install enablement for oem-enabled devices via update manager:
https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/1908050

While this works great for some situations, it can lead to users unexpectedly installing the oem meta package + associated kernel, overwriting an existing fips installation, as the "Improved hardware support" bundle may not be noticed when operating update-manager

[Expected Behavior]
For non linux-generic running installs, the post-install oem enablement functionality should not trigger, nor should it add the additional repositories to the client's sources.list.d.

[Observed Behavior]
sources.list.d is updated and "Improved hardware support" is allowed as an option in update-manager, which leads to clients unexpectedly losing compliance in fips environments.

[Replication Steps]
(Using Dell Inc. Precision 7920 Tower/060K5C)
1. Install from current focal ISO
2. Attach a ua subscription
3. Enable the fips-updates service
4. Reboot the system, login the desktop and wait for a while. The notification will pop up and it will show "Improved hardware support" on the certified machines that has the OEM metapackage support.
5. Click through the update-manager prompt and install the oem packages
6. Reboot check fips status

oem's config in /etc/default/grub.d/* does not have a number prefix, and thus will always override 99-ubuntu-fips.cfg when calling update-grub.

Revision history for this message
Kyler Hornor (kylerhornor) wrote :
summary: - Post-Install enablement of OEM-enabled devices will overwrite FIPs
+ FIPS/OEM installation compatibility is unclear to the end-user
description: updated
Revision history for this message
Dan Bungert (dbungert) wrote :

re Subiquity:
> Messaging should clearly indicate that installing the oem packages will make the environment incompatible with fips/RT kernel/ etc.

I would like to see a more precise list of these incompatibilities, as I am not familiar with them.
Who can provide that information?

Changed in subiquity (Ubuntu):
status: New → Incomplete
Revision history for this message
Kyler Hornor (kylerhornor) wrote :

The issue arises with any package leveraging the GRUB_FLAVOUR_ORDER cfg. While originally requested by the FIPS team to ensure the env didn't accidentally boot to -generic, it was later implemented by the oem packages to ensure the oem kernel was preferred. The fact that it prefers oem when both are installed is only due to the cfg file names.

Subiquity just offers to install the oem packages if ubuntu-drivers finds a match on the hardware. The issue here is "Hey would you like to install some special hardware enablement drivers?" isn't necessarily indicative that you would no longer be able to install fips. This should more clearly convey what's happening here.

This issue has a capture of the some of the pages I'm talking about:
https://github.com/canonical/subiquity/pull/1196

Revision history for this message
Dan Bungert (dbungert) wrote :

That is going to be a really hard label to write.
Not in terms of the code change, that's the easy part, how does one explain to end users that some GRUB_FLAVOUR_ORDER kernels are OK and some aren't? GRUB_FLAVOUR_ORDER is not a detail many users want to think about, they just want hardware to work.

If this problem were more limited in scope we could say something like "Third-party drivers should not be installed on systems that will be used for FIPS or the real-time kernel." But as is this is too open-ended - if we phrase this incorrectly it is going to sound like every kernel is a problem.

Revision history for this message
Kyler Hornor (kylerhornor) wrote :

> "Third-party drivers should not be installed on systems that will be used for FIPS or the real-time kernel."

This may be sufficient. We're already talking about a fairly extreme edge case here. The footprint of hardware that triggers the ubuntu-drivers criteria that also is intended to use FIPS later is already quite small. If we ever add additional packages that leverage GRUB_FLAVOUR_ORDER in the future, we could add information as needed?

The subiquity piece of this is likely the smallest part. The `aggressive` installs of the oem packages in already-configured desktop instances via update-manager is the bigger headache.( i.e. running fips fine, then oem packages `randomly` install, and I lose compliance without being aware.)

Revision history for this message
Dan Bungert (dbungert) wrote :
Changed in subiquity (Ubuntu):
status: Incomplete → In Progress
assignee: nobody → Dan Bungert (dbungert)
Revision history for this message
Lucas Albuquerque Medeiros de Moura (lamoura) wrote :

We discussed this on the UA team and we believe we can warn the user of the potential problems of enabling FIPS we using a OEM kernel, but we don't think we should block this on the UA side, since users could still be aware of the risk and follow through with the procedure.

However, before we deliver this through UA, we believe we need to sort the GRUB_FLAVOUR_ORDER scenario. Otherwise we will warn the user and even if they want to proceed with FIPS, the will end up in the OEM kernel regardless.

But let us know if you think we should definitely block enabling FIPS on a machine running a OEM kernel.

Revision history for this message
Kyler Hornor (kylerhornor) wrote :

I'd (personally) be much more on board with a warning vs a hard block as well.

The GRUB_FLAVOUR_ORDER issue was met with some resistance by both the FIPs team and the OEM team. From what I gathered, GRUB_FLAVOUR_ORDER was originally implement for the FIPS kernel, but later leveraged by OEM.

The OEM supersede was preferred by the OEM team, as OEM packages are intended to be temporary (and only for critical functionality issues) until adopted upstream, after which they are intended to be removed (?).

The argument from the FIPs side more leans towards the `forced` OEM installs. This falls outside of the UA client's responsibility though, as it can trigger before or after ua enablement via ubuntu-drivers-common.

I'd agree that we'd want a definitive decision on preferred preference before pulling the trigger on the UA changes though.

Revision history for this message
Dan Bungert (dbungert) wrote :

Is there a future where we can remove this warning?

The version of Subiquity on Jammy has some differences in the driver screen, so that's done here.
https://github.com/canonical/subiquity/pull/1369

Changed in subiquity (Ubuntu):
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers