Comment 7 for bug 1954456

I would really prefer not to revert this, and it will break F35 again. I'm still quite convinced updating /etc/default/grub is the best way to set kernel options.

I'm looking at

https://logserver.rdoproject.org/13/36713/6/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/b2300e8/logs/undercloud/home/zuul/overcloud-hardened-uefi-full.log.txt.gz

I see it setting options @

2021-12-10 14:01:15.309 | + echo GRUB_DEVICE=LABEL=img-rootfs
...

and then

2021-12-10 14:01:15.314 | + grub2-mkconfig -o /boot/grub2/grub.cfg

should update the entries. I added the BLS dumping after this to confirm in the original change.

However, what we see there is kernel args that do not match what has been set in /etc/default/grub @

2021-12-10 14:01:17.124 | args="ro console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"

Indeed I opened up the qcow2 https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-20210830.0.x86_64.qcow2 and this is what is set

><fs> cat /etc/default/grub
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

That .qcow2 image has an entry for a79a1f2ff34249debb171ba47537dc23-5.14.0-0.rc7.54.el9.x86_64.conf which we see in the entries. I would expect that to not be updated as the machine-id doesn't match.

The one below that (id="4a199ba39c6a452e8bf4840dfb1393c7-5.14.0-25.el9.x86_64") I do not see in the image. It suggests to me that grub2-mkconfig is making this entry?

The question is, why is this not obeying the /etc/default/grub settings in this situation. We know this *does* work -- https://4e6c3bd4754e70179f24-822effae95fb7ef10a1069deb9c06d04.ssl.cf2.rackcdn.com/821652/2/check/dib-nodepool-functional-openstack-centos-9-stream-src/49be447/nodepool/builds/test-image-0000000002.log is an example. There we do exactly the same thing in the gate builds; see @

2021-12-14 06:37:47.788 | + grub2-mkconfig -o /boot/grub2/grub.cfg

and note the dump after that shows the kernel options specified correctly.

We need to understand what is misconfigured so that grub2-mkconfig does not work. I am quite convinced that we should not be working around grub2-mkconfig not working -- we should be setting up the system so that we are in-line with it's expectations. Otherwise, we ship images that don't work.