Change to mount sequence order breaks persistence on casper-rw partitions

Bug #1489855 reported by Thomas Weissel on 2015-08-28
This bug affects 19 people
Affects Status Importance Assigned to Milestone
casper (Ubuntu)

Bug Description

the system boots fine when using a casper-rw FILE but drops to a busybox when using a partition

the short log would be:
Begin: Running /scripts/casper-premount ... done.
umount: can't umount /cdrom: Device or resource busy
Warning: Unable to find the persistent home medium
umount: can't umount /cdrom: Device or resource busy
Warning: Impossible to include the casper-sn Snapshot
umount: can't umount /cdrom: Device or resource busy
Warning: Impossible to include the home-sn Snapshot

removing the "persistence" keyword from the syslinux.cfg works and the live usb drive boots just fine.
using a persistence file instead of the partition and the usb drive boots just fine.

i found older bug reports concerning the same problem but the fix proposed in 2010 is already integrated into the script "casper".

so i started the flashdrive with the casper debug= option and i see the following (after probing several other devices it finally finds the right device and partition (sdb3)


+ cow_backing_mp=/cdrom
+ [ -e /cdrom/casper-rw ]
+ umount /cdrom
+ sys2dev /sys/block/sdb/sdb3
+ sysdev=/block/sdb/sdb3
+ udevadm info -q name -p /block/sdb/sdb3
+ echo /dev/sdb3
+ devname=/dev/sdb3
+ /sbin/blkid -s LABEL -o value /dev/sdb3
+ [ casper-rw = casper-rw ]
+ echo /dev/sdb3
+ return
+ cowprobe=/dev/sdb3
+ [ -b /dev/sdb3 ]
+ cowdevice=/dev/sdb3
+ get_fstype /dev/sdb3
+ local FSTYPE
+ local FSSIZE
+ fstype
+ eval FSTYPE=ext4 FSSIZE=8458862592
+ FSTYPE=ext4 FSSIZE=8458862592
+ [ ext4 != unknown ]
+ echo ext4
+ return 0
+ cow_fstype=ext4
+ cow_mountopt=rw,noatime
+ mount -t ext4 -o rw,noatime /dev/sdb3 /cow
+ [ ! -d /cow/upper ]
+ mkdir -p /cow/upper
+ continue
+ continue
+ mkdir -p /cow/work
+ [ -f /cow/format ]
+ modprobe -q -b overlay
+ grep -q ^overlay$
+ cut -f2 /proc/filesystems
+ UNIONFS=overlay
+ break

this looks fine to me.. it looks like it recognizes everything .. it's ext4 .. label casper-rw.. it's mounting it...

a little bit further down in the loooong log file it states the following:


+ cow_backing_mp=/home-rw-backing
+ [ -e /home-rw-backing/home-rw ]
+ umount /home-rw-backing
+ sys2dev /sys/block/sdb/sdb3
+ sysdev=/block/sdb/sdb3
+ udevadm info -q name -p /block/sdb/sdb3
+ echo /dev/sdb3
+ devname=/dev/sdb3
+ /sbin/blkid -s LABEL -o value /dev/sdb3
+ [ casper-rw = home-rw ]
+ get_fstype /dev/sdb3
+ local FSTYPE
+ local FSSIZE
+ fstype
+ eval FSTYPE=ext4 FSSIZE=8458862592
+ FSTYPE=ext4 FSSIZE=8458862592
+ [ ext4 != unknown ]
+ echo ext4
+ return 0
+ [ ext4 = vfat ]
+ homecow=
+ [ -b ]
+ [ n != y ]
+ log_warning_msg Unable to find the persistent home medium
+ _log_msg Warning: Unable to find the persistent home medium\n
+ [ n = y ]
+ printf Warning: Unable to find the persistent home medium\n
Warning: Unable to find the persistent home medium

a warning about the home medium is shown in both cases (file and partition) but the persistence file works never the less..

my system:

kubuntu linux 15.10 beta1

uname -a
Linux wald 4.1.0-3-generic #3-Ubuntu SMP Tue Jul 28 12:25:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

JoseStefan (josestefan) wrote :

Did you ever resolve this?

I think I have the same issue with the final release of Ubuntu 15.10
ece816e12f97018fa3d4974b5fd27337 *ubuntu-15.10-desktop-amd64.iso

Here's what I've done so far, I've combined concepts of different guides. I'm starting from a Windows 10 environment.
1) I used diskpart on the windows command line to wipe the partition table off the usb drive, As window's "Disk Management" can be restrictive (next step). Any good partition tool would do.
2) Created a 1.2 GB fat32 partition, 4096 cluster size. The rest of the usb is left unpartitioned. I determined the size by trial and error, originally I used a guide that suggest installing on the whole usb and resizing/shrinking, I found this to be easier.
3) Used the tool at pendrive linux. Added about 70mb of persistence.

Until this point everything works as expected. It Boots to the live environment, it has persistence and I'm using only a portion of the USB.

4) I created two ext4 partitions. My USB drive is 16GB, and wanted to have more control of what would be in and out of casper-rw So the first one is labeled "casper-rw" and the second one is labeled "data".

At this point I can reboot, and everything is just like before, the system will continue to use the casper FILE, and not take advantage of the PARTITION. Both partitions are automatically mounted on /media/ubuntu

It's when I delete the casper-rw file that the problem happens. After a few failed attempts, I decided to move it out instead. I'm using windows for this, and it sits on my desktop while I test. I can then revert successfully.

When I boot without the casper-rw FILE, I get a busybox, even though I have the casper-rw PARTITION created.

I expect it to automatically revert to using the partition.

JoseStefan (josestefan) wrote :

I repeated my same steps with 15.04 and the casper-rw PARTITION works.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in casper (Ubuntu):
status: New → Confirmed
syscon-hh (syscon-kono) wrote :

My interim solution to use '--- persistent' on a separate partition:

1. *rename* the partition 'casper-rw' into *system*

2. add a file 'casper-rw' of *0 Byte' into first/main *vfat* partition

3. add a new partition *home-rw* (optional)

4. on UEFI-computers add *persistent* to the first menuentry of the */boot/grub/grub.cfg*

Now start the usb-device up to *busybox* appears, then type to prompt:

1. blkid -> identify partition 'system' -> exemplary -> '/dev/sdb2'

2. mount /dev/sdb2 /cow -> ignore messages

3. exit

The live session will start and save all changes made to the live session - have fun.

What I'm locking for will be a solution such as integrated entry into the preseed-files or ???

erasmo52 (erasmo-is) wrote :

Hi, i experienced the same problem. I tried also to install the new kubuntu 16.04 (xenial) and it have the same bug.
The interim way proposed by syscon-hh worked for me too, but we need to find a better normal way to fix the problem.
Probably there must be something to modify in one of the scripts Handling the boot process at the init-top level.
There is somebody out there who can study the case and propose a solution?
Where is the kubuntu staff? Why this bug still unassigned to somebody.
I am trying to study the case, but .... it would be better if somebody more skilled than me could do it.
Anyway thanks to everybody.

Andrew Tapia (catgaming) wrote :

This also appears to be a problem with the final release of Lubuntu 15.10. Taking (basically) the same steps as JoseStefan yields the same results.

Pelle Johnsen (pelle-johnsen) wrote :

Ran into the same issue with Ubuntu 16.04 beta, trying syscon-hh's workaround.

Pelle Johnsen (pelle-johnsen) wrote :

The workaround also works for me .. but it is very tedious to have to go through these steps on every boot. I have tried getting the casper teams attention here:

JoseStefan (josestefan) wrote :

Note that this used to work on 15.04 and is broken for 15.10 and seems like 16.04 too.
So maybe someone can diff the changes to see what part of the code is likely to cause this?

Even though the original bug report is for Kubuntu, I experienced the bug with regular Ubuntu. Don't know if the bug needs to be updated to reflect that. Also don't know if all the other flavors would be affected.

JoseStefan (josestefan) wrote :

Tested with the 14.04.4 LTS (trusty), and also got a busy box.
807fa1f246b719d28d0b362fd2f31855 *ubuntu-14.04.4-desktop-amd64.iso

Compared the change logs, still haven't looked at the source (programming not my thing). They have this change in common:
casper (1.340.2) trusty; urgency=low
casper (1.363) wily; urgency=low

* scripts/casper: migrate existing persistent disk images to the updated upper/work form. Record the actual format used and ensure we use the existing format and abort if not possible. (LP: #1481117)

I guess my next test is to find the older Ubuntu 14.04.3 iso and test with that, as it doesn't have that change implemented, maybe as far back as 14.04.2

JoseStefan (josestefan) wrote :

While performing the steps I indicated before:
* Ubuntu 14.04.3 failed (busybox)
* Ubuntu 14.04.2 worked as expected (df confirms correct disk space)

so in theory it was working with
casper (1.340)

and broken with
casper (1.340.1) trusty; urgency=low

but 15.04 (vivid) is working with:
casper (1.360) vivid; urgency=medium
which already implements (1.340.1) as (1.347 and 1.348)

So I don't know what to make of that.

Pelle Johnsen (pelle-johnsen) wrote :

I also tested with elementary os 0.3.2 (which is 14.04 based). This was working fine.

Max (max1975) wrote :

Hi to all,

I actually found the relevant difference between a working 15.04 and a not working 16.04. In the function setup_unionfs () in scripts/casper (located in the ramdisk) the sequence of mounting changed from root partition then persistent partition (15.04) to 1st persistent then root partition (16.04) for whatever reason. If you change back (simple cut and paste the section) to the sequence of 15.04 and build then a new ramdisk you can run 16.04 with a persistent partition.

br max

Thomas Weissel (xapient) wrote :


could you please post a more precise description (line numbers from > to)
i just edited the file


then executed

sudo update-initramfs -u

and now i'm rebuilding my life system (this takes a while)

hopefully i interchanged the right lines.. (i only had a 14.04 casper file as reference and the differences are quite big)

Thomas Weissel (xapient) wrote :

the patched setup_unionfs() function

Max (max1975) wrote :

Apparently you already figured it out by yourself. Have fun.

Joel Ong (joel-ong) on 2016-09-05
summary: - kubuntu 15.10 beta1 live usb drops to busybox with persistence PARTITION
+ Change to mount sequence order breaks persistence
summary: - Change to mount sequence order breaks persistence
+ Change to mount sequence order breaks persistence on casper-rw
+ partitions
Thomas Weissel (valueerror) wrote :

ok... i ran into this bug AGAIN and i was dumb enough to think that this would definitely be resolved by now (since the bugfix is already provided here) i lost hours with debugging until i finally realized that this bug is still there..

could you please fix this ? it's 2 minutes work ... copy and paste ! and casper is not usable in persistent mode.. this bug is critical !

Vecdi Burak Bengi (burakbengi) wrote :

please fix this bug with already! proposed fix. this bug is (let's say it one more time) CRITICAL!

g (garethic) wrote :

Does this mean I can, say, take the squashfs from a broken Kubuntu 16 live usb & replace the aquashfs of a working kubuntu 14 usb? Or will that break everything else in sight?
Because, I am NFI on how/where to find Casper.

Molto congrats on finding the solution!

Thomas Weissel (xapient) wrote :

well.. no.. if you change the squashfs file you replace the whole system.. you will then have ubuntu 14.04 and NOT 16.04

just use the information from post #14 (the path to the file in question) and replace the faulty version of the function setupunionfs() with the one i attached to post #15


Pankaj Mohan (proaudience) wrote :

There is a workaround for avoiding this bug when creating such pen-drives. You should create three partitions : 1) A fat32 formatted 350MB for storing 'boot' and 'EFI' folders, 2) An ext4 formatted 2GB for storing the rest of Ubuntu's ISO content & 3) An ext4 formatted and labelled as 'casper-rw' in whatever size remains available for persistence. Finally enter 'set root=(hd0,2)' (without single quotes) in grub.cfg and the drive starts working as per your wishes.

One could do the same even with two partitions, provided they were ready to use the ISO file itself instead of its extracted content. I've described all this in my blog post linked below. Remember, this is a layman's perspective, and written by gathering information through a trial and error approach, so kindly don't expect any technical explanations for whatever I've been able to list down there...

EvilSupahFly (seann-giffin) wrote :

On ext4, the journal is written to the same location over and over again, which can have a seriously negative impact on the lifespan of the flash memory. If you want your USB stick to last longer, consider using ext2 instead because it doesn't use journaling. Otherwise, well done Pankaj!

JoseStefan (josestefan) wrote :

I've been tracking this bug for a few years now. For a bug of such HEAT shouldn't it by now at least have an importance other than UNDECIDED? And shouldn't the bug have enough information to be TRIAGED instead of CONFIRMED. It just seems this bug is not moving forward and just going to sit in it's current state forever.

For every new Ubuntu release, I'm afraid to invest more time trying to get persistence to work, because I've already dedicated much time to this subject in the past, and have personally failed. Instead I come back to his bug to check the status before even trying. So I'm not sure of the status of the bug for current releases of Ubuntu.

I'm creating these USB sticks from Windows. There are some workaround posted here, like Pankaj's. But those steps require that you already be booted on Ubuntu to begin with. So that would at least require creating 2 USB sticks if you are starting from Windows. And like I said, I don't want to invest more time and resources on the subject than I have already. Would be nice if the Windows tools were updated with his observations and created an alternate persistence structure. But that's another subject.

Thomas Weissel (xapient) wrote :

this is really disappointing...

   mount: mounting /cow on /root falied: invalid argument overlay mount failed

i just changed the lines in /usr/share/initramfs-tools/scripts/casper and the live system booted without any problems...

could you please finally fix this bug !!

@seann-giffin i thought the same thing and therefore changed to ext2 - unfortunately it happens a lot that the flashdrives were removed in an unfortunate way ... we had a lot of filesystem errors... it seemed that ext4 was more healthy on the long run

DC-THINK (libratwo) wrote :

look into the code, u can also see the fault posted debug log above:

when using a USB HDD(/dev/sdb) partition /cdrom(/dev/sdb1 vfat) casper-rw(/dev/sdb2 ext2)

1. mount /dev/sdb1 /cdrom (/casper is exist(will load filesystem.squashfs lately))
2. find casper-rw(file and partition in same time(cause bug))
   casper.setup_unionfs-> casper-helper.find_cow_device
   for each /dev/sdb*
      a. find file
          a.1 /dev/sdb* not mounted(mount /dev/sdb* to /casper-rw-backing)
          a.2 /dev/sdb1 mounted(remount -rw on /cdrom)
          then check the casper-rw file's existence
          ***after check(not exist), it will umount the /dev/sdb*, for /cdrom will be umounted, here is bug ***
      b. find partition just check LABEL = casper-rw
    in order of /dev/sdb1 /dev/sdb2, so /cdrom be umounted

workaround may be:
  1. put casper-rw(/dev/sdb1) before /cdrom(/dev/sdb2) partition
  2. or modify find_cow_device, 1st find partiton, 2nd find file(don't do in the same time)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Related questions