Unable to mount lvmcache root device at boot time

Bug #1423796 reported by MegaBrutal
138
This bug affects 26 people
Affects Status Importance Assigned to Milestone
initramfs-tools (Ubuntu)
Medium
Unassigned
linux (Ubuntu)
Medium
Andy Whitcroft
lvm2 (Ubuntu)
High
Unassigned

Bug Description

I'm experimenting with Vivid Vervet on a virtual machine and tried a new LVM feature, lvmcache. I made a cache for the root file system, rebuilt the initrd and rebooted the VM.

At boot time, the system failed to activate the root LV. After some investigation, I found out, it's because the initrd is missing some essential stuff needed for activating a cached LV.

The initrd was missing the dm-cache module. I regenerated the initrd with explicitly listing dm-cache in /etc/initramfs-tools/modules, but the system still can't boot up, because now it is missing the /usr/sbin/cache_check utility.

As SSDs are becoming more and more common, I think it will be common to use them as cache for root file systems, thus it is mandatory to make sure that an initrd can mount an lvmcached root device when necessary, preferably without /etc/initramfs-tools/modules and other manual initrd hacking.

System details:

Linux lvmvm 3.18.0-13-generic #14-Ubuntu SMP Fri Feb 6 09:55:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Distributor ID: Ubuntu
Description: Ubuntu Vivid Vervet (development branch)
Release: 15.04
Codename: vivid

LVM version: 2.02.111(2) (2014-09-01)
Library version: 1.02.90 (2014-09-01)
Driver version: 4.28.0

Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1423796

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
MegaBrutal (qbu6to) wrote :

This is an lvm2 bug and I wanted to report it to lvm2, not Linux. I remember that I exactly typed lvm2, I don't know why it was still reported to linux. Could someone help me to sort it out?

affects: linux (Ubuntu) → lvm2 (Ubuntu)
Changed in lvm2 (Ubuntu):
status: Incomplete → New
MegaBrutal (qbu6to)
tags: added: vivid
description: updated
description: updated
MegaBrutal (qbu6to)
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in lvm2 (Ubuntu):
status: New → Confirmed
Changed in lvm2 (Ubuntu):
importance: Undecided → High
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
status: New → Confirmed
Changed in initramfs-tools (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Changed in linux (Ubuntu):
importance: Undecided → Medium
assignee: nobody → Andy Whitcroft (apw)
milestone: none → ubuntu-15.03
Revision history for this message
George Moutsopoulos (gmoutso) wrote :

I also tried the instructions in
http://forums.debian.net/viewtopic.php?f=5&t=119644
but still cannot reboot into cached hdd system

removing the cache (lvremove lvcache) every time I fail to boot, by using a rescue usb, gave me some corruption and was just inconvenient to further experiment. so I ended up giving up until someone else succeeds using lvmcache.

My use would be a laptop with a small ssd 24GB drive and I think this would be a cool way to speed up the system. actually it was intended for that and the MS windows system did that before I wiped it out. Setting up the lvm and installing ubuntu was easy, setting up the lvmcache was easy too... but couldn't manage to make it boot :/

Revision history for this message
George Moutsopoulos (gmoutso) wrote :

MegaBrutal, can you give some instructions. I followed the instructions at
 http://forums.debian.net/viewtopic.php?f=5&t=119644
but still couldn't boot

Revision history for this message
George Moutsopoulos (gmoutso) wrote :

OK, I managed! I had to add more modules

sudo echo "dm_cache" >> /etc/initramfs-tools/modules
sudo echo "dm_cache_mq" >> /etc/initramfs-tools/modules
sudo echo "dm_persistent_data" >> /etc/initramfs-tools/modules
sudo echo "dm_bufio" >> /etc/initramfs-tools/modules

plus, add cache_check as in
 http://forums.debian.net/viewtopic.php?f=5&t=119644

Booted!

This is how I set up the lvmcache
sudo vgextend LinuxVG /dev/sdb2
sudo lvcreate -L 10G -n rootCacheLV LinuxVG /dev/sdb2
sudo lvcreate -L 24M -n rootCacheMetaLV LinuxVG /dev/sdb2
sudo lvcreate -L 10G -n homeCacheLV LinuxVG /dev/sdb2
sudo lvcreate -L 24M -n homeCacheMetaLV LinuxVG /dev/sdb2
sudo lvconvert --type cache-pool --poolmetadata LinuxVG/rootCacheMetaLV --cachemode writethrough LinuxVG/rootCacheLV
sudo lvconvert --type cache-pool --poolmetadata LinuxVG/homeCacheMetaLV --cachemode writethrough LinuxVG/homeCacheLV
sudo lvconvert --type cache --cachepool LinuxVG/rootCacheLV LinuxVG/rootLV
sudo lvconvert --type cache --cachepool LinuxVG/homeCacheLV LinuxVG/homeLV

where LinuxVG has the root at rootLV and home at homeLV and sdb2 is an SSD

Revision history for this message
Adrian Wilkins (adrian-wilkins) wrote :

Thank you George! Your contribution was the final thing I needed to get this to work :

http://askubuntu.com/questions/620480/how-to-install-ubuntu-with-both-disk-encryption-and-ssd-caching

Revision history for this message
Paul TREHIOU (paul-trehiou) wrote :

I have the same problem with Ubuntu 15.04
The solution of gmoutso worked thanks !

Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-15.03 → ubuntu-15.07
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-15.07 → ubuntu-15.08
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-15.08 → ubuntu-15.09
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-15.09 → ubuntu-15.10
Revision history for this message
Paul TREHIOU (paul-trehiou) wrote :

Is this corrected in the 15.10 ?

Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-15.10 → ubuntu-15.11
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-15.11 → ubuntu-15.12
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-15.12 → ubuntu-16.01
Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-16.01 → ubuntu-16.02
Revision history for this message
Jared Stemen (jstemen) wrote :

I'm running Ubuntu 15.10 and I was unable to get gmoutso's solution working.

Andy Whitcroft (apw)
Changed in linux (Ubuntu):
milestone: ubuntu-16.02 → ubuntu-16.03
Revision history for this message
Jared Stemen (jstemen) wrote :

Correction, I was able to get gmoutso's solution to work. I realized that I needed to create a separate boot partition that didn't use lvm caching. In retrospect, this makes a lot of sense.

Revision history for this message
Jérôme Poulin (jeromepoulin) wrote :

After making my system work again, which took a while since most live media and even the installer don't support LVM cache, I had it working for writeback on my system with the following.

/etc/initramfs-tools/hooks/lvm2_cache
{{{
#!/bin/sh

PREREQ="lvm2"

prereqs()
{
    echo "$PREREQ"
}

case $1 in
prereqs)
    prereqs
    exit 0
    ;;
esac

if [ ! -x /sbin/lvm ]; then
    exit 0
fi

. /usr/share/initramfs-tools/hook-functions

copy_exec /usr/sbin/cache_check
copy_exec /usr/sbin/cache_dump
copy_exec /usr/sbin/cache_metadata_size
copy_exec /usr/sbin/cache_repair
copy_exec /usr/sbin/cache_restore

for x in dm_cache_cleaner dm_cache_smq dm_cache dm_persistent_data dm_bio_prison
do
    manual_add_modules ${x}
done
}}}

Revision history for this message
Jérôme Poulin (jeromepoulin) wrote :

That new file could be added to the thin-provisioning-tools which would trigger an initramfs update.

I forgot to remove cache_dump, cache_restore and cache_metadata_size which are not needed to boot.

Revision history for this message
Astara (astara) wrote :

@George:
George Moutsopoulos (gmoutso) wrote on 2015-04-29:
> OK, I managed! I had to add more modules
>
> sudo echo "dm_cache" >> /etc/initramfs-tools/modules
> sudo echo "dm_cache_mq" >> /etc/initramfs-tools/modules
> sudo echo "dm_persistent_data" >> /etc/initramfs-tools/modules
> sudo echo "dm_bufio" >> /etc/initramfs-tools/module
---
Not exactly related to the original bug, but wanted to address a boo-boo in the above.
sudo echo "string" won't use root's access/permissions to write to the files when using the shell-redirection operators '>' or '>>'. sudo will execute 'echo' with root privs, and that will echo "dm_cache" (et al.), however, that output is redirected by the *SHELL* (running as
"whoever is running the shell") -- usually the user.

To get around this problem (and not saying there might not be any easier ways, as this seemed a bit roundabout to get what I wanted (have root write the output to "wherever"), I used
"dd", as in:

echo "dm_cache"|sudo dd of=destination status=none

 or to append:

echo "dm_cache"|sudo dd status=none oflag=append conv=notrunc of=/etc/initramfs-tools/modules

 or to do all at once:

(echo dm_cache
echo dm_cache_mq
echo dm_persistent_data
echo dm_bufio)|sudo dd status=none oflag=append conv=notrunc of=/initramfs-tools/modules

I wanted to write a single "char" to a file in /proc/sys/vm (drop_caches).

Perhaps needless to say, I put it in a script file to save on typing. :-)

Revision history for this message
RNZ (renoizer) wrote :

Same bug in 17.04: configure lvmcache and reboot - system can't map cached device to /dev/mapper/
Fixed by update initramfs with hook in upper comment: https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/1423796/comments/12

Revision history for this message
Johannes Bauer (johannesbauer) wrote :

I just ran into this with a cleanly installed 17.04. Figured it out myself, added "copy_exec /usr/sbin/cache_check" to /usr/share/initramfs-tools/hooks/lvm2 in order to fix it.

Can we PLEASE get this bug -- which renders systems UNBOOTABLE after attaching a cache -- fixed, especially since there are good and reasonable patches available already (#12)?

Revision history for this message
Bernd Schubert (aakef) wrote :

Related to debian bug #773731

Brad Figg (brad-figg)
tags: added: cscc
Revision history for this message
Andrew Berry (andrewberry) wrote :

This is still broken in Ubuntu 18.04.

Revision history for this message
elatllat (elatllat) wrote :

yah I have actually never seen a bug fixed on the Ubuntu bug tracker, just deal with it upstream or use a different distribution.
IMO Ubuntu should delete bugs.launchpad.net and just merge pull requests on github/etc.

Revision history for this message
Rhys (rhvs) wrote :

Still a problem in 20.04...

Revision history for this message
Carlo Vallati (cvalla) wrote :

I confirm, still there in 20.04

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers