ubiquity btrfs install fails to boot (grub rescue> prompt)

[Michael Vogt]

Binary package hint: ubiquity

After a install of natty desktop from today (02-02-2011) the install finished without a issue but the subsequent reboot fails to boot.

error: file not found
grub rescue>

[Michael Vogt (mvo)]

Trying to interact with the grub-rescue> is a pain, insmod (hd0,msdos1)/@/boot/grub/normal.mod gives me a error: file not found and without readline support its slow to type commands.

[Colin Watson (cjwatson)]

Sounds fun. I'll have a go at reproducing and debugging this.

[Colin Watson (cjwatson)]

'set prefix=(hd0,msdos1)/@/boot/grub; insmod normal; normal' gets me up to a better prompt after various error messages, but 'root' is still wrong. In any case, this points to a problem with subvolume support, which we'll need to fix properly.

[Michael Vogt (mvo)]

Thanks a lot Colin! It seems like the grub part of the problem might be that is simply search the wrong path. When copying "/boot" to the root part of the FS (instead of @/boot) it works better. A symlink works as well. But I also see a bunch of error message (like error: sparse file not allowed).

I'm in busybox land after grub works, but to fix that is easy, just adding "rootflags=subvol=@" to the kernel commandline. This does give me a booting system (no X, but that appears to be a seperate issue).

[Colin Watson (cjwatson)]

I'd rather avoid symlink hackery. Fixing $prefix isn't too difficult,
but fixing $root is hard because there are some bits of the GRUB
configuration that assume it's a bare drive name. I'm going to have to
talk with upstream to see what we ought to do here; it's possible that
we need to extend the drive name syntax to be able to specify
subvolumes.

Don't subvolumes have UUIDs too? It would help if we could pass the
subvolume UUID to GRUB's 'search' command; it might be easier that way
than trying to figure out how to add a suffix.

(Alternatively, we could use 'btrfs subvolume set-default'. I don't
know if that would be any less confusing.)

The reason you have no X might well be because / (in the @ subvolume) is
mode 0700. I've corrected this in partman-btrfs in bzr.

[Colin Watson (cjwatson)]

It seems to get further if I run 'btrfs subvolume set-default 256 /mnt' (where the number came from 'btrfs subvolume list /mnt'), but GRUB prints an error "sparse file not allowed" (which is from 'loadenv') and then hangs.

[Colin Watson (cjwatson)]

... but if you get past that, then mounting subvol=@home tries to mount /@/@home rather than just /@home, and the subvolume name is only allowed to be a single pathname component, so that isn't going to work.

[Colin Watson (cjwatson)]

Actually, the fundamental problem here may be that grub-mkrelpath includes the subvolume path in its output, but *only sometimes*, and not when we need it to. It looks as though fixing that, plus adding suitable rootflags= as Michael points out, may be enough.

[Colin Watson (cjwatson)]

I've fixed the core of this (grub-mkrelpath) in the relevant upstream branch. I'm working on a couple of other things to nail this down properly:

  * grub-probe seems unhappy after booting into the target system, although oddly it seemed fine while chrooted
  * add rootflags=
  * figure out why we're getting "sparse file" errors from load_env and fix them

[Colin Watson (cjwatson)]

The grub-probe problem was pilot error; I forgot to run it as root. Sorry for the noise.

I've changed GRUB to add rootflags=subvol=<name> automatically where needed, and have uploaded partman-btrfs to fix subvolume permissions. Nearly there ...

[Colin Watson (cjwatson)]

The "sparse file" errors are a different problem. It's actually a misleading error message: what's happening is that GRUB's btrfs implementation doesn't implement the file read hook interface for returning blocklists to calling code. I posted to grub-devel about this and the upstream maintainer pointed out that, even aside from multi-device problems, writing to btrfs from GRUB is fundamentally risky because:

 * the same block may be used by multiple snapshots
 * every tree which uses a given block will contain its checksum, and so on recursively

However, btrfs reserves space at the start for the boot loader. This space is more than GRUB needs to embed itself, and so we could use 1KB of it for an environment block.

In any case, this is not a new problem that arose from using subvolumes, nor does it prevent booting (you get a spurious "Press any key to continue" prompt, but if you just ignore it it'll boot anyway). Thus I'm going to disregard it for the purposes of closing this bug, since I believe I've committed fixes for all the other problems, and track the environment block problem separately.

[Id2ndR (id2ndr)]

@Michael Vogt, I use the kernel option "rootflags=subvol=@" as you and get the same problem. I solved it by booting in recovery mode and running "chmod +rx /" (it should be related to the subvolume permissions of partman-btrfs that Colin Watson tall about).

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 1.99~rc1-2ubuntu1

---------------
grub2 (1.99~rc1-2ubuntu1) natty; urgency=low

  * Resynchronise with Debian. Remaining changes:
    - Adjust for default Ubuntu boot options ("quiet splash").
    - Default to hiding the menu; holding down Shift at boot will show it.
    - Set a monochromatic theme and an aubergine background for Ubuntu.
    - Apply Ubuntu GRUB Legacy changes to legacy update-grub script: title,
      recovery mode, quiet option, tweak how memtest86+ is displayed, and
      use UUIDs where appropriate.
    - Fix backslash-escaping in merge_debconf_into_conf.
    - Remove "GNU/Linux" from default distributor string.
    - Add crashkernel option.
    - Bypass menu unless other OSes are installed or Shift is pressed.
    - Allow Shift to interrupt 'sleep --interruptible'.
    - Reduce visual clutter in normal mode.
    - Remove verbose messages printed before reading configuration.
    - Suppress kernel/initrd progress messages, except in recovery mode.
    - Handle filesystems loop-mounted on file images.
    - Ignore devices loop-mounted from files in Linux grub.d scripts.
    - Show the boot menu if the previous boot failed.
    - Don't generate device.map during grub-install or grub-mkconfig.
    - Adjust upgrade version checks for Ubuntu.
    - Suppress "GRUB loading" message unless Shift is held down.
    - Adjust versions of grub-doc and grub-legacy-doc conflicts.
    - Fix LVM/RAID probing in the absence of /boot/grub/device.map.
    - Look for .mo files in /usr/share/locale-langpack first.
    - Make sure GRUB_TIMEOUT isn't quoted unnecessarily.
    - Build-depend on qemu-kvm rather than qemu-system for grub-pc tests.
    - Use qemu rather than qemu-system-i386.
    - Add a grub-rescue-efi-amd64 package.
    - On Wubi, don't ask for an install device, but just update wubildr
      using the diverted grub-install.
    - Enable lua from grub-extras.
    - Incorporate the bitop library into lua.
    - Add enum_pci function to grub module in lua.
    - Check hardware support before using gfxpayload=keep.
    - Build part_msdos and vfat into EFI boot images.
    - Put second and subsequent Linux menu entries in a submenu.
    - Preferred resolution detection for VBE.
    - Set vt.handoff=7 for smooth handoff to kernel graphical mode.
    - Don't ask where to install if upgrading and core.img doesn't exist.
    - Drop the default priority of grub2/linux_cmdline to medium.
  * Fix quoting glitch in ubuntu_gfxpayload_filter.patch.

grub2 (1.99~rc1-2) unstable; urgency=low

  * Merge 1.98+20100804-13 and 1.98+20100804-14, updating translations:
    - Kazakh (Baurzhan Muftakhidinov / Timur Birsh).
  * mkconfig_skip_dmcrypt.patch: Refer to GRUB_PRELOAD_MODULES rather than
    suggesting people write a /etc/grub.d/01_modules script (thanks, Jordan
    Uggla).
  * Handle empty dir passed to grub_find_root_device_from_mountinfo; fixes
    grub-mkrelpath on btrfs subvolumes (LP: #712029).
  * Add rootflags=subvol=<name> if / is on a btrfs subvolume (LP: #712029).
  * Upload to unstable.
 -- Colin Watson <email address hidden> Tue, 08 Feb 2011 12:42:13 +0000

[David Nielsen (davidnielsen-deactivatedaccount)]

The subvolume permissions still seem to be problematic according to testers on the forums using the feb. 12 or 13 daily.

http://ubuntuforums.org/showpost.php?p=10454087&postcount=8

[Colin Watson (cjwatson)]

ubiquity hasn't been reuploaded yet, so permissions on desktop installs
will still be wrong. I'll upload that today.

[AO (aofrl10n)]

On Natty x64 installed from the 18/02/2011 daily (in the last three dailies, Ubiquity locks on the first screen), /boot installed on it's own 250 MB btrfs partition, / on it's own 19 GB btrfs partition, system fully updated from main server as of 13:23 UTC today, the sparse file not allowed error message is still present so this has not been fixed, though the system remains bootable.

[Colin Watson (cjwatson)]

Alain-Olivier, I explained the "sparse file not allowed" message in comment 12.

[Ken D'Ambrosio (ken-jots)]

btrfs still isn't working as a boot destination. Indeed, when I try to use it, I usually throw a grub error during install (and, obviously, boot). The part that puzzles me the most is that, when I mount the partition, it seems that all the root-level files and directories are in a subdirectory named "@" (sans quotes) -- I assume there's some installer magic involved here. As I read that it *had* been working in alpha-2 (with some minor glitches), I was hoping that the beta would have addressed the issues. Installing grub by hand doesn't throw errors at install time, but still doesn't boot.

Note: this is for 32-bit server beta1.

[Colin Watson (cjwatson)]

The @ directory is a subvolume. Even if you don't need subvolumes at
installation time (although the installer does create them in some
cases, in order to support cunning tricks by update-manager later on),
the way btrfs works is that you need to create at least one subvolume
up-front otherwise you have to completely reorganise your filesystem if
you ever need to use the feature in the future.

You can use 'mount -o subvol=@' to mount the appropriate subvolume.

I'll give this another try in the coming week. The annoying thing is
that GRUB's btrfs support works *some* of the time, and has stubbornly
refused to fail when I'm trying to debug it. Could you please file a
new bug on the grub2 package in Ubuntu (don't search for duplicates -
I'd prefer a fresh bug report in this case), and describe exactly how
you reproduced this bug? Ideally, I'd like something as close as
possible to a recipe that I can use in a virtual machine.

[Id2ndR (id2ndr)]

rootflags=subvol=@ seems not to be added anymore with latest grub update.

[Colin Watson (cjwatson)]

On Sun, Apr 03, 2011 at 11:09:36AM -0000, Id2ndR wrote:
> rootflags=subvol=@ seems not to be added anymore with latest grub
> update.

That would be a consequence of this same bug, although I would ask that
you please use bug 732149 rather than continuing to comment on this
closed bug report.