Most recent grub update uses root=/dev/sdX for boot image, may fail to boot

[Daniel Smedegaard Buus]

Hello :)

The recent update (today) of grub,
   grub-pc:amd64 (1.99-21ubuntu3, 1.99-21ubuntu3.1)
   grub-pc-bin:amd64 (1.99-21ubuntu3, 1.99-21ubuntu3.1)
   grub-common:amd64 (1.99-21ubuntu3, 1.99-21ubuntu3.1)
killed my grub, apart from also looking strange (used to be gray background, and a nice logo splash when booting, now black background and just low-res text when booting).

The BOOT_IMAGE command defined root as - in my case - /dev/sdq2, while at boot it would be /dev/sdb2. Also, this might not stay the same regardless, and as I understand it, this should point to either a partition UUID or perhaps /dev/disk/by-id/something-or-other?

In any event, as it stands, all boot options ends with busybox and an initramfs prompt. Editing the grub command and fixing the root= to the right path before booting will send me normally into Kubuntu where I can (hopefully) fix the problem manually.

Cheerio :)

Daniel

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: xorg 1:7.6+12ubuntu1
ProcVersionSignature: Ubuntu 3.0.0-17.30-generic 3.0.22
Uname: Linux 3.0.0-17-generic x86_64
NonfreeKernelModules: nvidia zfs zcommon znvpair zavl zunicode
ApportVersion: 2.0.1-0ubuntu8
Architecture: amd64
Date: Sat Jun 2 10:44:50 2012
InstallationMedia: Kubuntu 12.04 LTS "Precise Pangolin" - Beta amd64 (20120421.1)
ProcEnviron:
 LANGUAGE=en_US:en
 TERM=xterm
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: xorg
Symptom: display
UpgradeStatus: No upgrade log present (probably fresh install)

[Steve Langasek (vorlon)]

Reassigning to grub2, which is the source package for grub-pc.

> The BOOT_IMAGE command defined root as - in my case - /dev/sdq2,

There is no "BOOT_IMAGE" variable anywhere is the standard grub configuration, so I have no idea what this refers to. And grub will only use /dev/sd* style device names for booting if you specify 'GRUB_DISABLE_LINUX_UUID=true' in /etc/default/grub.

Please attach your /etc/default/grub and /boot/grub/grub.cfg from the affected system.

[Daniel Smedegaard Buus (danielbuus)]

Sorry for being unclear. With BOOT_IMAGE, I was referring to the output of cat /proc/cmdline, e.g.
BOOT_IMAGE=/@/boot/vmlinuz-3.0.0-17-generic root=/dev/sdq2 ro rootflags=subvol=@ quiet splash vt.handoff=7.

Above is what grub creates on my system with this is /etc/default/grub:

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

So either GRUB_DISABLE_LINUX_UUID is now defaulting to true, or something else is going on.

Actually, it seems allright in the generated /boot/grub/grub.cfg (see attached).

I do note, however, the "set root='(hd16,msdos2)'" bit in grub.cfg. This corresponds exactly to sdq2 with hd0 being sda. It wasn't in grub.cfg I saw the "root=sdq2", as at that point I wasn't able to boot the system (root drive being sda), it was when editing the command line in the GRUB interface.

This may be a wildly inaccurate bug report in that this behavior may either be the default(?) or it may be much older than the most recent update. It just happend to coincide with me doing hard drive fiddling and updates to GRUB rolling in.

[Steve Langasek (vorlon)]

On Mon, Jun 04, 2012 at 07:39:17AM -0000, Daniel Smedegaard Buus wrote:
> Sorry for being unclear. With BOOT_IMAGE, I was referring to the output of
> cat /proc/cmdline, e.g.
> BOOT_IMAGE=/@/boot/vmlinuz-3.0.0-17-generic root=/dev/sdq2 ro rootflags=subvol=@ quiet splash vt.handoff=7.

> Above is what grub creates on my system with this is /etc/default/grub:

> # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
> #GRUB_DISABLE_LINUX_UUID=true

> So either GRUB_DISABLE_LINUX_UUID is now defaulting to true, or
> something else is going on.

Something else is going on. Nobody else has reported this behavior, and
nothing changed in the precise-updates version that would cause the behavior
to change.

I really don't know where that 'hd16' has come from, but that may be the
root of your problem.

> This may be a wildly inaccurate bug report in that this behavior may
> either be the default(?) or it may be much older than the most recent
> update. It just happend to coincide with me doing hard drive fiddling
> and updates to GRUB rolling in.

Yes, I think it's not tied to the recent update - if you were fiddling with
hard drives at the same time, that was probably the trigger.

[Phillip Susi (psusi)]

Your grub.cfg appears to be using the UUID correctly, are you sure /proc/cmdline still isn't showing it? Can you run this script and attach the generated file?

http://sourceforge.net/projects/bootinfoscript/

[Daniel Smedegaard Buus (danielbuus)]

Attached.

/proc/cmdline shows sdq2 (which IS correct — though only as long as I don't add or remove drives, swap cables, USB HDDs, etc. :) )

[Phillip Susi (psusi)]

Can you attach your /var/log/kern.log file?

[Daniel Smedegaard Buus (danielbuus)]

There you go :)

[Phillip Susi (psusi)]

That seems to be a few days out of date. Could you reboot the system and check /proc/cmdline again? Since grub.cfg is not passing that device, the only way I can see for it to be there is if grub.cfg was changed, and you have not rebooted since.

[Daniel Smedegaard Buus (danielbuus)]

Yes, but it's the most recent boot. I won't be able to reboot before in about four days (jobs running), but I can assure you nothing has changed since then so it'll boot the same :)

[Daniel Smedegaard Buus (danielbuus)]

FWIW, prior to that boot I redid update-grub and update-initramfs in the hops it'd "fix" something. Everything stayed the same.

Just to make sure we're on the same page: /dev/sdq2 *IS* my root partition (normally, and right now). It's not that sdq2 was fetched out of thin air.

On my MacBook Air with a freshly installed Kubuntu 12.04, the behavior is the same, although here root=/dev/mapper/sda6_crypt, both in GRUB itself and /proc/cmdline.

I don't mean to sound wise-assy, but did you try cat'ing /proc/cmdline on your own install? Or hitting 'e' in GRUB to see what root= says there?

Cheers :)

[Phillip Susi (psusi)]

The cryptsetup isn't the same at all, since it specifies the crypt device in grub.cfg, and that is what you get on the kernel command line. Whatever grub.cfg says to pass is what gets passed and thus, what shows up in /proc/cmdline, which is why this isn't making any sense. I suppose when you reboot, it wouldn't hurt to hit 'e' and check... if it says sdq2 there instead of the UUID, then that would explain why /proc/cmdline is not as expected. The only way that it could differ at that point from what the cfg file says is if you somehow have a different grub install that isn't using that cfg file.

[Daniel Smedegaard Buus (danielbuus)]

Well, it was exactly doing 'e' in GRUB that revealed the problem it first. This is also where I manually edited it at the time from sdq2 to sdb2 (IIRC) to be able to boot. Sorry if I'm unclear in my explanations. Bottomline: hitting 'e' shows sdq2.

The only place I can see in the grub cfg that doesn't use a UUID is the "set root='(hd16,msdos2)'" part I noted earlier. And after an update-grub, it's all still the same.

Your /proc/cmdline shows a UUID for root?

[Phillip Susi (psusi)]

Can you run:

sudo mount /dev/sdq2 /mnt
ls /mnt/boot
ls /mnt/@/boot

[Daniel Smedegaard Buus (danielbuus)]

Well, since it's the root fs, I can't mount it twice, but I already have the btrfs root volume set up in fstab to be mounted in /mnt/btrfs, so it looks like this:

daniel@lnxsrv:~$ sudo mount /mnt/btrfs
daniel@lnxsrv:~$ ls /mnt/btrfs/@/boot/
abi-3.0.0-17-generic config-3.2.0-24-generic initrd.img-3.2.0 memtest86+_multiboot.bin vmcoreinfo-3.0.0-17-generic
abi-3.2.0-24-generic grub initrd.img-3.2.0-24-generic System.map-3.0.0-17-generic vmlinuz-3.0.0-17-generic
config-3.0.0-17-generic initrd.img-3.0.0-17-generic memtest86+.bin System.map-3.2.0-24-generic vmlinuz-3.2.0-24-generic

FYI, in fstab:
UUID=aa16e554-8ba2-4537-8bc6-a2c37f160369 /mnt/btrfs btrfs defaults,noatime,noauto,ssd,subvolid=0 0 0

[Daniel Smedegaard Buus (danielbuus)]

Could me being with kernel 3.0 still have anything to do with it? (Don't see why personally, but then again, there's a lot of things I don't know ;) )

[Daniel Smedegaard Buus (danielbuus)]

It's weird, I just installed Ubuntu in a VM on my Air, and it doesn't show this at all. It uses UUID just fine. Exact same GRUB version. It can't be the sheer number of drives in my desktop triggering it? Like if your root volume is on drive > 16... Doesn't make sense, though :-/

[Phillip Susi (psusi)]

Now what about ls /mnt/btrfs/boot?

What I suspect is that you have two different /boot directories, one at /boot, and one at /@/boot, which is what you see mounted in /boot. The latter copy says to use the UUID, but grub is actually using the former copy, which must say not to.

[Daniel Smedegaard Buus (danielbuus)]

Ah, that's why you asked. I thought it was a typo, as there's only @ and @home, and snapshots thereof:

drwxr-xr-x 1 root root 302 Jun 2 12:36 @/
drwxr-xr-x 1 root root 204 Apr 23 13:35 @-2012-04-23.1438/
drwxr-xr-x 1 root root 218 Apr 23 14:44 @-2012-04-23.1553-manual/
drwxr-xr-x 1 root root 252 Apr 23 16:12 @-2012-04-23.1719-zfs/
drwxr-xr-x 1 root root 252 Apr 23 16:12 @-2012-04-23.1742-nvidia-and-messages-logging/
drwxr-xr-x 1 root root 252 Apr 23 23:03 @-2012-04-24.0010-misc/
drwxr-xr-x 1 root root 252 Apr 23 23:03 @-2012-04-24.1904-misc/
drwxr-xr-x 1 root root 302 May 6 10:30 @-2012-05-20.1731/
drwxr-xr-x 1 root root 302 May 6 10:30 @-2012-05-28.0926/
drwxr-xr-x 1 root root 46 Apr 25 12:13 @home/
drwxr-xr-x 1 root root 32 Apr 23 13:00 @home-2012-04-23.1442-initial-not-yet-logged-in/
drwxr-xr-x 1 root root 46 Apr 24 10:47 @home-2012-04-24.1904-misc/

[Phillip Susi (psusi)]

Well I'm about stumped then. Can you edit /boot/grub/grub.cfg and add a dummy parameter to the kernel command line, reboot, press e at the grub menu and see if the dummy parameter shows up?

[Daniel Smedegaard Buus (danielbuus)]

Hi, sorry for the very long delay (holiday! yay!) :)

Returning this weekend, I decided to get rid of my secondary NVIDIA card and use this box solely as an HTPC machine (at least GUI-wise, it's still a server otherwise ;) ).

Tons of updates, and I removed my SII3114 PCI card and replaced it with a PCIe-based 3132 one now that I had the vacant PCIe slot.

During this, a lot of drives changed ports, including the system one. No problems occurred.

So, I cannot say if this was due to an upgrade, if my problem was just some weird cosmic ray hitting me, or what, but there's no problem anymore.

Sorry to keep you busy, and thanks for everything. I'm marking this invalid.

Cheers,
Daniel