Unable to mount lvmcache root device at boot time

[MegaBrutal]

I'm experimenting with Vivid Vervet on a virtual machine and tried a new LVM feature, lvmcache. I made a cache for the root file system, rebuilt the initrd and rebooted the VM.

At boot time, the system failed to activate the root LV. After some investigation, I found out, it's because the initrd is missing some essential stuff needed for activating a cached LV.

The initrd was missing the dm-cache module. I regenerated the initrd with explicitly listing dm-cache in /etc/initramfs-tools/modules, but the system still can't boot up, because now it is missing the /usr/sbin/cache_check utility.

As SSDs are becoming more and more common, I think it will be common to use them as cache for root file systems, thus it is mandatory to make sure that an initrd can mount an lvmcached root device when necessary, preferably without /etc/initramfs-tools/modules and other manual initrd hacking.

System details:

Linux lvmvm 3.18.0-13-generic #14-Ubuntu SMP Fri Feb 6 09:55:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Distributor ID: Ubuntu
Description: Ubuntu Vivid Vervet (development branch)
Release: 15.04
Codename: vivid

LVM version: 2.02.111(2) (2014-09-01)
Library version: 1.02.90 (2014-09-01)
Driver version: 4.28.0

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1423796

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[MegaBrutal (qbu6to)]

This is an lvm2 bug and I wanted to report it to lvm2, not Linux. I remember that I exactly typed lvm2, I don't know why it was still reported to linux. Could someone help me to sort it out?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[George Moutsopoulos (gmoutso)]

I also tried the instructions in
http://forums.debian.net/viewtopic.php?f=5&t=119644
but still cannot reboot into cached hdd system

removing the cache (lvremove lvcache) every time I fail to boot, by using a rescue usb, gave me some corruption and was just inconvenient to further experiment. so I ended up giving up until someone else succeeds using lvmcache.

My use would be a laptop with a small ssd 24GB drive and I think this would be a cool way to speed up the system. actually it was intended for that and the MS windows system did that before I wiped it out. Setting up the lvm and installing ubuntu was easy, setting up the lvmcache was easy too... but couldn't manage to make it boot :/

[George Moutsopoulos (gmoutso)]

MegaBrutal, can you give some instructions. I followed the instructions at
 http://forums.debian.net/viewtopic.php?f=5&t=119644
but still couldn't boot

[George Moutsopoulos (gmoutso)]

OK, I managed! I had to add more modules

sudo echo "dm_cache" >> /etc/initramfs-tools/modules
sudo echo "dm_cache_mq" >> /etc/initramfs-tools/modules
sudo echo "dm_persistent_data" >> /etc/initramfs-tools/modules
sudo echo "dm_bufio" >> /etc/initramfs-tools/modules

plus, add cache_check as in
 http://forums.debian.net/viewtopic.php?f=5&t=119644

Booted!

This is how I set up the lvmcache
sudo vgextend LinuxVG /dev/sdb2
sudo lvcreate -L 10G -n rootCacheLV LinuxVG /dev/sdb2
sudo lvcreate -L 24M -n rootCacheMetaLV LinuxVG /dev/sdb2
sudo lvcreate -L 10G -n homeCacheLV LinuxVG /dev/sdb2
sudo lvcreate -L 24M -n homeCacheMetaLV LinuxVG /dev/sdb2
sudo lvconvert --type cache-pool --poolmetadata LinuxVG/rootCacheMetaLV --cachemode writethrough LinuxVG/rootCacheLV
sudo lvconvert --type cache-pool --poolmetadata LinuxVG/homeCacheMetaLV --cachemode writethrough LinuxVG/homeCacheLV
sudo lvconvert --type cache --cachepool LinuxVG/rootCacheLV LinuxVG/rootLV
sudo lvconvert --type cache --cachepool LinuxVG/homeCacheLV LinuxVG/homeLV

where LinuxVG has the root at rootLV and home at homeLV and sdb2 is an SSD

[Adrian Wilkins (adrian-wilkins)]

Thank you George! Your contribution was the final thing I needed to get this to work :

http://askubuntu.com/questions/620480/how-to-install-ubuntu-with-both-disk-encryption-and-ssd-caching

[Paul TREHIOU (paul-trehiou)]

I have the same problem with Ubuntu 15.04
The solution of gmoutso worked thanks !

[Paul TREHIOU (paul-trehiou)]

Is this corrected in the 15.10 ?

[Jared Stemen (jstemen)]

I'm running Ubuntu 15.10 and I was unable to get gmoutso's solution working.

[Jared Stemen (jstemen)]

Correction, I was able to get gmoutso's solution to work. I realized that I needed to create a separate boot partition that didn't use lvm caching. In retrospect, this makes a lot of sense.

[Jérôme Poulin (jeromepoulin)]

After making my system work again, which took a while since most live media and even the installer don't support LVM cache, I had it working for writeback on my system with the following.

/etc/initramfs-tools/hooks/lvm2_cache
{{{
#!/bin/sh

PREREQ="lvm2"

prereqs()
{
    echo "$PREREQ"
}

case $1 in
prereqs)
    prereqs
    exit 0
    ;;
esac

if [ ! -x /sbin/lvm ]; then
    exit 0
fi

. /usr/share/initramfs-tools/hook-functions

copy_exec /usr/sbin/cache_check
copy_exec /usr/sbin/cache_dump
copy_exec /usr/sbin/cache_metadata_size
copy_exec /usr/sbin/cache_repair
copy_exec /usr/sbin/cache_restore

for x in dm_cache_cleaner dm_cache_smq dm_cache dm_persistent_data dm_bio_prison
do
    manual_add_modules ${x}
done
}}}

[Jérôme Poulin (jeromepoulin)]

That new file could be added to the thin-provisioning-tools which would trigger an initramfs update.

I forgot to remove cache_dump, cache_restore and cache_metadata_size which are not needed to boot.

[Astara (astara)]

@George:
George Moutsopoulos (gmoutso) wrote on 2015-04-29:
> OK, I managed! I had to add more modules
>
> sudo echo "dm_cache" >> /etc/initramfs-tools/modules
> sudo echo "dm_cache_mq" >> /etc/initramfs-tools/modules
> sudo echo "dm_persistent_data" >> /etc/initramfs-tools/modules
> sudo echo "dm_bufio" >> /etc/initramfs-tools/module
---
Not exactly related to the original bug, but wanted to address a boo-boo in the above.
sudo echo "string" won't use root's access/permissions to write to the files when using the shell-redirection operators '>' or '>>'. sudo will execute 'echo' with root privs, and that will echo "dm_cache" (et al.), however, that output is redirected by the *SHELL* (running as
"whoever is running the shell") -- usually the user.

To get around this problem (and not saying there might not be any easier ways, as this seemed a bit roundabout to get what I wanted (have root write the output to "wherever"), I used
"dd", as in:

echo "dm_cache"|sudo dd of=destination status=none

 or to append:

echo "dm_cache"|sudo dd status=none oflag=append conv=notrunc of=/etc/initramfs-tools/modules

 or to do all at once:

(echo dm_cache
echo dm_cache_mq
echo dm_persistent_data
echo dm_bufio)|sudo dd status=none oflag=append conv=notrunc of=/initramfs-tools/modules

I wanted to write a single "char" to a file in /proc/sys/vm (drop_caches).

Perhaps needless to say, I put it in a script file to save on typing. :-)

[RNZ (renoizer)]

Same bug in 17.04: configure lvmcache and reboot - system can't map cached device to /dev/mapper/
Fixed by update initramfs with hook in upper comment: https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/1423796/comments/12

[Johannes Bauer (johannesbauer)]

I just ran into this with a cleanly installed 17.04. Figured it out myself, added "copy_exec /usr/sbin/cache_check" to /usr/share/initramfs-tools/hooks/lvm2 in order to fix it.

Can we PLEASE get this bug -- which renders systems UNBOOTABLE after attaching a cache -- fixed, especially since there are good and reasonable patches available already (#12)?