18.04LTS upgrade of grub-common:amd64 FAILs in post-install; incorrectly REWRITES user's /etc/default/grub

[pgnd]

upgrading packages, as usual, on a long-time-in-production Ubuntu 1804LTS server instance,

 lsb_release -rd
  Description: Ubuntu 18.04.4 LTS
  Release: 18.04
 uname -rm
  5.3.0-40-generic x86_64

today's update included grub,

 cat /var/log/apt/history.log

  Start-Date: 2020-03-19 08:58:38
  Commandline: apt-get upgrade
  Upgrade: grub-common:amd64 (2.02-2ubuntu8.14, 2.02-2ubuntu8.15), grub2-common:amd64 (2.02-2ubuntu8.14, 2.02-2ubuntu8.15), grub-pc:amd64 (2.02-2ubuntu8.14, 2.02-2ubuntu8.15), grub-pc-bin:amd64 (2.02-2ubuntu8.14, 2.02-2ubuntu8.15)
  Error: Sub-process /usr/bin/dpkg returned an error code (1)
  End-Date: 2020-03-19 09:01:52

the update FAILs in grub post-processing

apparently, the install is -- new behavior -- REWRITING, incorrectly, the 'GRUB_CMDLINE_LINUX=' stanza in /etc/default/grub -- my _local_ end-user config

before

 cat /etc/default/grub
  ...
  GRUB_CMDLINE_LINUX_DEFAULT=""
 53 GRUB_CMDLINE_LINUX="rd.shell rd.udev.log_priority=info rd.auto=1 \
   dolvm lvmwait=/dev/mapper/VG0_ROOT \
   rootfstype=ext4 rootdelay=90 rootflags=journal_checksum \
   noresume pcie_aspm=off max_loop=256 transparent_hugepage=never clocksource=tsc clocksource_failover=tsc tsc=reliable force_tsc_stable=1 \
   scsi_mod.use_blk_mq=1 \
   net.ifnames=1 biosdevname=0 \
   showopts noquiet \
   log_buf_len=16M print_fatal_signals=1 systemd.log_level=info systemd.log_target=kmsg earlyprintk=vga,keep \
   mitigations=auto spec_store_bypass_disable=on \
  "
  ...

exec an update, as usual

 apt-get update && apt-get upgrade
  Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease
  Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease
  Hit:3 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease
  Hit:4 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease
  Reading package lists... Done
  Reading package lists... Done
  Building dependency tree
  Reading state information... Done
  Calculating upgrade... Done
  0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
  1 not fully installed or removed.
  After this operation, 0 B of additional disk space will be used.
  Do you want to continue? [Y/n]
 -> Y
  Do you want to continue? [Y/n] Y

now, fails

  Setting up grub-pc (2.02-2ubuntu8.15) ...
  Sourcing file `/etc/default/grub'
  /etc/default/grub: line 53: unexpected EOF while looking for matching `"'
  dpkg: error processing package grub-pc (--configure):
   installed grub-pc package post-installation script subprocess returned error exit status 2
  Errors were encountered while processing:
   grub-pc
  E: Sub-process /usr/bin/dpkg returned an error code (1)

it's incorrectly re-written /etc/default/grub

 cat /etc/default/grub
  ...
  GRUB_CMDLINE_LINUX_DEFAULT=""
 53 GRUB_CMDLINE_LINUX="rd.shell rd.udev.log_priority=info rd.auto=1 dolvm lvmwait=/dev/mapper/VG0_ROOT rootfstype=ext4 rootdelay=90 rootflags=journal_checksum noresume pcie_aspm=off max_loop=256 transparent_hugepage=never clocksource=tsc clocksource_failover=tsc tsc=reliable force_tsc_stable=1 scsi_mod.use_blk_mq=1 net.ifnames=1 biosdevname=0 showopts noquiet log_buf_len=16M print_fatal_signals=1 systemd.log_level=info systemd.log_target=kmsg earlyprintk=vga,keep mitigations=auto spec_store_bypass_disable=on"
   dolvm lvmwait=/dev/mapper/VG0_ROOT \
   rootfstype=ext4 rootdelay=90 rootflags=journal_checksum \
   noresume pcie_aspm=off max_loop=256 transparent_hugepage=never clocksource=tsc clocksource_failover=tsc tsc=reliable force_tsc_stable=1 \
   scsi_mod.use_blk_mq=1 \
   net.ifnames=1 biosdevname=0 \
   showopts noquiet \
   log_buf_len=16M print_fatal_signals=1 systemd.log_level=info systemd.log_target=kmsg earlyprintk=vga,keep \
   mitigations=auto spec_store_bypass_disable=on \
  "

prepending the EXISTING 'GRUB_CMDLINE_LINUX=' with a duplicate of itself ... plus an errant trailing `"'

fyi,

apt-cache policy grub2
grub2:
  Installed: (none)
  Candidate: 2.02-2ubuntu8.15
  Version table:
     2.02-2ubuntu8.15 500
        500 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
     2.02-2ubuntu8 500
        500 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

[C de-Avillez (hggdh2)]

This is a regression

Marking triaged, easily reproducible:

*BEFORE* dist-upgrade:

1. edit /etc/default/grub
2. change GRUB_COMLINE_LINUX_DEFAULT from:
     GRUBCMDLINE_LINUX_DEFAULT="quiet splash"
   to:
     GRUB_CMDLINE_LINUX_DEFAULT="quiet \"
splash"
3. run a dist-upgrade

Expected result: update-grub/install completes successfully

Actual result: update-grub fails (or apt fails), because /etc/default/grub now has a mangled GRUB_CMDLINE_LINUX_DEFAULT:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
splash"

I am setting importance to medium because it will only impact users with continuation lines.

[pgnd (pgnd)]

comment from hggdh in #irc,

'the last change on install was for 2.02-2ubuntu8.13'

re: impact ... sure, NO-continuation-lines-used is the likely majority

but, impact varies.

fwiw, here anyway, our internal prio's a bit higher.
100+ ubu servers with continuation lines in their grub cfg ...

in reply to:

Q: your servers, have they been there for a while (specifically, were they configured this way *before* grub 2.02-2ubuntu8.13?

A: yes, they've all been in place for years ... upgrading across multiple LTS releases etc. and, YES -- *all* my grub configs, not just these Ubu servers, use continuation lines.

[pgnd (pgnd)]

in comments from ubuntu-devel

 [17:50] <juliank> Continuation lines seem wrong to have in that file
 [17:50] <juliank> If you add a continuation line between 8.14 and 8.15, you can remove the continuation line again
 [17:51] <juliank> A future update should probably handle that, but a continuation line here is not expected, and there's no point rolling it back as that's a non default unexpected thing

as stated clearly at

 https://www.gnu.org/software/grub/manual/grub/html_node/Simple-configuration.html#Simple-configuration-handling

 ...
 The file /etc/default/grub controls the operation of grub-mkconfig. It is sourced by a shell script, and so must be valid POSIX shell input; normally, it will just be a sequence of ‘KEY=value’ lines, but if the value contains spaces or other special characters then it must be quoted
 ...

per POSIX

 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02_01

  Escape Character (Backslash)

   A <backslash> that is not quoted shall preserve the literal value of the following character, with the exception of a <newline>. If a <newline> follows the <backslash>, the shell shall interpret this as line continuation. The <backslash> and <newline> shall be removed before splitting the input into tokens. Since the escaped <newline> is removed entirely from the input and is not replaced by any white space, it cannot serve as a token separator.

all that's to say that continuation lines are, and have been, supported in posix-shell-compliant /etc/default/grub

that it's "non default unexpected thing" is specious ... at _best_ it's unused/untested by the pacakgers.

fwiw, with grub2-2.04 on numerous _other_ systems there are no problems whatsoever with these continuation lines; and Ubu 18LTS didn't either ... until 'recently'.

no, i don't yet have an identifying bisect ...

also, it's been commented that current bug is 'non-destructive'.

that's open to interpretation.

modifying an end-user config file persistently, and without user-interaction/-confirmmation ... such that a subsequent (re)upgrade/(re)install doesn't fix the problem, but rather requires a manual edit/fix of the config in order qualifies as 'destructive'. here, anyway.

[Julian Andres Klode (juliank)]

This bug report is missing data. The reproducer given is invalid, as it escapes the quoted string, as opposed to the newline. It also does not show that this is a regression in the 8.15 upload, as nothing in that upload changed.

The last change to logic here was made in 8.13, so any such bug must have hit before in the 8.14 upload. If you want to argue that this is not true, please provider a reproducer that does not trigger for 8.13 -> 8.14, but does trigger for 8.14 -> 8.15.

Regardless of what the POSIX spec says, this is highly unusual abuse of POSIX shell semantics, and I do not see a need to support it, so if it comes to it, my solution might just be to skip the optimisation in such cases, causing more conffile prompts for such (IMHO broken files).

It's highly advised to not use advanced features just because you can. KISS.

[Julian Andres Klode (juliank)]

I think the fix here is to fix the info manual to state that it should not normally be assignment lines, but must always be assignment lines, and maybe abort hard if we encounter exotic syntax like that.

[pgnd (pgnd)]

> IMHO broken files

please clarify under what circumstances you think it's appropriate/OK to modify user-config files, breaking them?

because that's what's being done here.

the installer is overwriting the /etc/default/grub file.

> highly unusual abuse of POSIX shell semantics,

horesefeathers

> I do not see a need to support it

ok, that's, then, the issue here.

if that's policy, then that's good to know/understand.
for _that_ there are clear/available solutions.

[Julian Andres Klode (juliank)]

Anyway, both

GRUB_CMDLINE_LINUX="moo
bar"

and

GRUB_CMDLINE_LINUX="foo \
bar"

break.

I have looked at OPs claim that this is a new regression, and verified that this is a wrong assertion - it was broken in 8.13:

ubuntu@boo1:~$ sudo apt install ./*.deb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'grub-common' instead of './grub-common_2.02-2ubuntu8.13_amd64.deb'
Note, selecting 'grub-efi-amd64-bin' instead of './grub-efi-amd64-bin_2.02-2ubuntu8.13_amd64.deb'
Note, selecting 'grub-efi-amd64' instead of './grub-efi-amd64_2.02-2ubuntu8.13_amd64.deb'
Note, selecting 'grub2-common' instead of './grub2-common_2.02-2ubuntu8.13_amd64.deb'
Suggested packages:
  multiboot-doc grub-emu xorriso desktop-base
The following packages will be upgraded:
  grub-common grub-efi-amd64 grub-efi-amd64-bin grub2-common
4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/3006 kB of archives.
After this operation, 35.8 kB of additional disk space will be used.
Get:1 /home/ubuntu/grub-efi-amd64_2.02-2ubuntu8.13_amd64.deb grub-efi-amd64 amd64 2.02-2ubuntu8.13 [47.8 kB]
Get:2 /home/ubuntu/grub-efi-amd64-bin_2.02-2ubuntu8.13_amd64.deb grub-efi-amd64-bin amd64 2.02-2ubuntu8.13 [654 kB]
Get:3 /home/ubuntu/grub2-common_2.02-2ubuntu8.13_amd64.deb grub2-common amd64 2.02-2ubuntu8.13 [533 kB]
Get:4 /home/ubuntu/grub-common_2.02-2ubuntu8.13_amd64.deb grub-common amd64 2.02-2ubuntu8.13 [1771 kB]
Preconfiguring packages ...
(Reading database ... 59745 files and directories currently installed.)
Preparing to unpack .../grub-efi-amd64_2.02-2ubuntu8.13_amd64.deb ...
Unpacking grub-efi-amd64 (2.02-2ubuntu8.13) over (2.02-2ubuntu8) ...
Preparing to unpack .../grub-efi-amd64-bin_2.02-2ubuntu8.13_amd64.deb ...
Unpacking grub-efi-amd64-bin (2.02-2ubuntu8.13) over (2.02-2ubuntu8) ...
Preparing to unpack .../grub2-common_2.02-2ubuntu8.13_amd64.deb ...
Unpacking grub2-common (2.02-2ubuntu8.13) over (2.02-2ubuntu8) ...
Preparing to unpack .../grub-common_2.02-2ubuntu8.13_amd64.deb ...
Unpacking grub-common (2.02-2ubuntu8.13) over (2.02-2ubuntu8) ...
Setting up grub-common (2.02-2ubuntu8.13) ...
Installing new version of config file /etc/grub.d/00_header ...
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
Setting up grub-efi-amd64-bin (2.02-2ubuntu8.13) ...
Setting up grub2-common (2.02-2ubuntu8.13) ...
Setting up grub-efi-amd64 (2.02-2ubuntu8.13) ...
/etc/default/grub: line 25: unexpected EOF while looking for matching ``'
/etc/default/grub: line 36: syntax error: unexpected end of file
Sourcing file `/etc/default/grub'
/usr/sbin/grub-mkconfig: 36: /etc/default/grub: Syntax error: EOF in backquote substitution
dpkg: error processing package grub-efi-amd64 (--configure):
 installed grub-efi-amd64 package post-installation script subprocess returned error exit status 2
Processing triggers for install-info (6.5.0.dfsg.1-2) ...
Processing triggers for systemd (237-3ubuntu10.39) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Errors were encountered while processing:
 grub-efi-amd64
E:...

[Julian Andres Klode (juliank)]

So given that 8.13 is affected - which was released a year ago, and it's a highly unusual configuration, this is not in need of an urgent hotfix.

[Dimitri John Ledkov (xnox)]

/etc/default/grub is ucf/debconf managed conffile, which has always been processed by grub2 maintainer scripts, and there was no change in behaviour of that in bionic.

You may not have ever noticed this before, because you might have only applied the config change without re-running dpkg-reconfigure on the relevant grub2 package, nor apply any updates that might be available. Note that on all new installations / launches of new instances, all updates are applied during installation process where possible. Thus this is not a regression in updates.

Many of the options specified there are not used by Ubuntu initramfs-tools and have no effect, why are you specifying dracut options? Have you converted your system from initramfs-tools to dracut?

Did you experience that the syntax used in the conffig file ever worked across grub2 upgrades or dpkg-reconfigure in any Ubuntu or Debian release?

Note, that you can always use dpkg preseed of the debconf key with all of the extra commandline options which will persist across grub2 updates; or use the .d drop in files (although those too, get parsed).

[pgnd (pgnd)]

as noted above, these configs have been in place " on a long-time-in-production Ubuntu 1804LTS server instance"

the configs have been in-place, with line breaks, for years. across countless upgrades, on ~100+ servers. including grub* packages.

upgrades have ONLY started failing with most recent upgrades.

could we have been 'just lucky'? sure. that in itself is disturbing & being investigated. likely to be migrated away from -- as these configs, with line beaks, are NOT a problem _anywhere_ 'else'.

is my specific config relevant? not if you accept,

  https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1868138/comments/1

as verification of this issue.

[C de-Avillez (hggdh2)]

"This bug report is missing data. The reproducer given is invalid, as it escapes the quoted string, as opposed to the newline."

Yes, it is broken, and it was my fault. I typed the lines instead of copy&paste. I apologise. But, as the poster themselves confirm just after this, it does break if the '\"´ is changed to '\'.

"It also does not show that this is a regression in the 8.15 upload, as nothing in that upload changed."

OK. It is a regression on the 8.13 upload. But it is STILL a regression. Or is it the rule that if we do not find a regression when a package is released, then it is not a regression anymore?