Curtin sneaks config into /etc/default/grub.d/

[Mark Shuttleworth]

Curtin puts content into /etc/default/grub.d/ but doesn't indicate that anything is wrong with /etc/default/grub

Since curtin is the authoritative installation, it can freely setup grub the way it believes it should look. At the moment, we end up with a very misleading picture, because /etc/default/grub looks authoritative but is completely and invisibly sidelined by /etc/default/grub.d/

So either just put the desired state into /etc/default/grub (leaving grub.d/ empty) or replace /etc/default/grub with a file that clearly indicates that the magic is now in /etc/default/grub.d/

[Junien F (axino)]

I reported something similar a while ago in LP#1527664

[Ryan Harper (raharper)]

Writing to /etc/default/grub can cause upgrade problems:

https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/901600

That includes either updating it to indicate that we've modified
the default configuration to match the environment or writing
out the deployed configuration directly, wholesale replacement.

I had suggested that maybe we could document this in packaged
version of /etc/default/grub in grub2, to indicate that variables
can be overridden by configurations in grub.d

https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1569567

In there, Mathieu has some alternatives; but it's not clear that any of
those handle all of the various situations we have.

1) cloud-images include a baked in console=ttyS0 which is highly useful
   for booting the images in VMs where we can have a serial all the time

2) in maas we re-use the cloud-images but real machines may or maynot have a ttyS0
   and so curtin adapted to always updating the grub config based on how
   maas booted the image;

3) to avoid upgrade issues, curtin did not modify /etc/default/grub directly

4) if curtin lets values from /etc/default/grub to be included in the configuration
   it writes (for example appending maas boot params in addition to what's built in
   to grub2) this may present conflicts; and maas/curtin own what that command
   line ought to be. Filtering isn't a solution either, curtin is only involved
   during deployment; if a dist-upgrade of grub2 introduces a new boot parameter
   it might affect the image booting.

Given the above scenarios I still believe my suggestion of documenting in the grub2 package version of /etc/default/grub that 'grub.d' directory may contain configurations that
override the values of /etc/default/grub seems the best course. This directs users who attempt to modify /etc/default/grub to check for contents in /etc/default/grub.d to see if their changes are shadowed, or rather to direct changes to those .d files.

[Mark Shuttleworth (sabdfl)]

On 20/10/16 17:30, Ryan Harper wrote:
> 3) to avoid upgrade issues, curtin did not modify /etc/default/grub
> directly

Does this avoid upgrade issues? Or does it make them inevitable?
Essentially what you're saying is "we don't want the user to be asked a
question during an upgrade". That's fine. But if there is stuff in
/etc/default/grub.d/ then whatever we change during upgrade in
/etc/default/grub is operating under the *delusion* that it is in
control. Therefor, all we're doing is making it harder for people to
*know* what the situation is, *especially* during an upgrade.

Or have I misunderstood what's going on?

Debian config file handling has this as a base problem, I accept that's
not a problem we created, but we need to grapple with it properly, not
kick the can elsewhere.

[Ryan Harper (raharper)]

On Fri, Oct 21, 2016 at 6:10 AM, Mark Shuttleworth <
<email address hidden>> wrote:

> On 20/10/16 17:30, Ryan Harper wrote:
> > 3) to avoid upgrade issues, curtin did not modify /etc/default/grub
> > directly
>
> Does this avoid upgrade issues? Or does it make them inevitable?
> Essentially what you're saying is "we don't want the user to be asked a
> question during an upgrade". That's fine. But if there is stuff in
>

That's the upgrade issue that I'm aware of which was the reason for the
change. Details are here:

https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/901600

> /etc/default/grub.d/ then whatever we change during upgrade in
> /etc/default/grub is operating under the *delusion* that it is in
> control. Therefor, all we're doing is making it harder for people to
> *know* what the situation is, *especially* during an upgrade.
>

> Or have I misunderstood what's going on?
>

Your understanding is correct. It currently _is_ harder to know what's
going
on because the current behavior is NOT documented in the configuration files
themselves.

Comments in the bug I linked to above confirm that as do other bugs being
filed related to user confusion.

> Debian config file handling has this as a base problem, I accept that's
> not a problem we created, but we need to grapple with it properly, not
> kick the can elsewhere.
>

Agreed.

> --
> You received this bug notification because you are subscribed to curtin.
> Matching subscriptions: curtin-bugs-all
> https://bugs.launchpad.net/bugs/1635181
>
> Title:
> Curtin sneaks config into /etc/default/grub.d/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1635181/+subscriptions
>

[MelkorLord (melkorlord)]

I'm glad I found this bug report! I've spent *a* *lot* of time trying to understand why my grub settings wheren't applied after a clean install of 18.04!

I was modifying /etc/default/grub as usual and nothing changed in the /boot/grub/grub.cfg, none of the options I was setting for the kernel command line appeared in the generated config after "update-grub". /etc/default/grub.d/* was overriding everything. I really cursed the guy who put there the configuration with hardcoded defaults that could *not* be overridden.

Well, I see I'm not the only one having trouble with that.

Put whatever you want in /etc/defaut/grub.d/* but /etc/default/grub should have the *final* word on the matter after "update-grub"!

[Ebbex (eb4x)]

Just to chime in, I wasted to much time trying to figure this out aswell.

[Nobuto Murata (nobuto)]

Cross-linking: https://bugs.launchpad.net/curtin/+bug/1527664

IMHO, writing /etc/default/grub.d/50-curtin.cfg as:
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT <curtin specific options if any>"
might work as a stop-gap solution so user modified /etc/default/grub will be also respected.

[Mark Eichin (eichin-gmail)]

I just spent 4+ hours trying to figure out why every bit of documentation (info, ask.ubunutu, etc) about booting multiple partitions wasn't working on a stock ubuntu-18.04.1-live-server-amd64.iso install. Only ended up finding it by putting a `set -x` in `/etc/grub.d/30_os-prober`, seeing it fail with `GRUB_DISABLE_OS_PROBER=true`, and doing a `find|grep` on all of `/etc` to find the culprit.

While you could fix the `maybe-ubiquity` bit by making it append, I'm not sure how you fix the os-prober part other than just not doing it in the first place? I'm guessing it simplifies things at install time, but is there a reason to silently (or at all) disable that feature once the initial install is done?

[Martin Widmark (pholostan)]

I'm sorry, but is this some kind of bad joke? I just spent some hours researching why my kernel parameters didn't update properly when editing /etc/default/grub on a stock ubuntu server 18.04.1 server install.

I had a /etc/default/grub.d/50-curtin-settings.cfg that messed things up. Seems to be undocumented and very unwelcome behavior.

[Stefan Alfredsson (alfs)]

I had the same experience as #8 and #9.

Some hours wasted troubleshooting why the kernel didn't have expected options, only to find "curtin the culprit", which led me to this bug report.

Here's the kicker - I didn't even have curtin installed! Why is the config file still present on the system?

# dpkg --list '*curtin*'
dpkg-query: no packages found matching *curtin*
# dpkg -S 50-curtin-settings.cfg
dpkg-query: no path found matching pattern *50-curtin-settings.cfg*
#
# grep curtin /var/lib/dpkg/info/*
# ls /etc/default/grub.d/50-curtin-settings.cfg
/etc/default/grub.d/50-curtin-settings.cfg

I suggest 'update-grub' at least report when variables are overrriden and where.

[Nelson Minar (g-nelson)]

Boy this bug is vexing. Another report of this bug has info that it was fixed in the Curtin sources in December 2018.
https://bugs.launchpad.net/curtin/+bug/1527664

[Mathieu Trudel-Lapierre (cyphermox)]

As an update to this:

I think we agreed (in a short meeting between people involved in curtin, cloud-init, grub2, etc.) that installers are indeed meant to be authoritative on writing /etc/default/grub.

One work item that came out of this was to at the very least make it clear what files are being used to generate grub.cfg. I've uploaded this to Disco already:

https://launchpad.net/ubuntu/+source/grub2/2.02+dfsg1-5ubuntu9

The same fixes are also being prepared for SRU in 18.04 and 18.10 (they should be available in -proposed incessantly).

I opened bug 1812863 since I didn't know this one was here -- regardless, there'd still be the question of handling config merging when grub2 expects to have distro-wide defaults, installers write their own things, and on top of that users might want to make configuration changes. This is still an issue we'll have to handle when curtin/cloud-init do write to /etc/default/grub directly, so this here bug can remain open with a grub2 task to denote the work that is needed there. We've already started looking into how to handle it, too.

For the time being, running 'update-grub' will provide additional information on what files are being sourced to generate the final grub.cfg configuration file that lands in /boot/grub/grub.cfg:

$ sudo update-grub
[sudo] password for mtrudel:
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg'
Generating grub configuration file ...
[...]
done

[Mate Kukri (mkukri)]

I've marked the cloud image variation of the same problem as a duplicate of this.

There is also a discussion scheduled for next week to sort out a way forward here.