Please do not go to release with this issue! This is a MAJOR regression compared to 19.10, and considering the amount of pain the previous persistent issue with LTS has caused (just go to reddit, superuser or askubuntu and search for "mounting /cow on /root/cow failed" reports to get an idea of just how many Ubuntu users are being negatively impacted by this), if Ubuntu does value its userbase, it does not want a repeat of the 18.04 persistence fiasco for the next LTS. I will provide steps on how to replicate the issue below, but first of all, I want to express my puzzlement with regards to unilaterally deciding to introduce a new 'writable' label for persistent partitions. Why???? "writable" provides no hints about persistence, and is way too generic to actually be understood. Unless you are using an optical media, where it's the hardware itself that is preventing write access, tagging a partition as "writeable" yields nothing, since, in the absolute, EVERY partition is of course writeable. So this tells nothing of value to users when it comes to persistence. Why not take a page from Debian here and just use 'persistence' for the label? At least, if people don't understand why their partition is called that way, they can perform a search and get a good overview of what persistent means. But with 'writable', you get nothing of the sort. And seriously, for crying out loud, could you please coordinate with other distros when it comes to persistence. It's bad enough that Ubuntu chose to use a different kernel option ('persistent' vs. Debian's 'persistence') but you had a golden opportunity to bridge a gap by following what Debian was doing (by introducing the new 'persistence' partition label), yet chose once again to disregard the common good and do it your way. Which brings me to the main point, the 'casper-rw' race condition is a MAJOR issue, which is far from being limited to slow machines. I am seeing it consistently happening on all of the UEFI platforms I am testing with, be it relatively slow dual-core based platform from >5 years ago to a recent 6-cores platform bought in January this year. So can you *PLEASE* test on real hardware instead of on VMs? I am confident that if you do test it on real hardware by creating a media in the manner highlighted below, you will consistently see the issue. Below are the exact commands I used to create a UEFI-bootable persistent USB Flash Drive, which you should be able to follow more or less exactly to replicate the problem (tested with the latest focal-desktop-amd64.iso daily build from 2020.04.03): ----------------------------------------------------------------------------------------- root@nano:/# ## Make sure to change the following disk to your USB media root@nano:/# export TARGET_DISK=/dev/sda root@nano:/# ## The following two commands erase the partition tables root@nano:/# dd if=/dev/zero of=$TARGET_DISK bs=512 count=34 34+0 records in 34+0 records out 17408 bytes (17 kB, 17 KiB) copied, 0.00602524 s, 2.9 MB/s root@nano:/# dd if=/dev/zero of=$TARGET_DISK bs=512 count=34 seek=$((`blockdev --getsz $TARGET_DISK` - 34)) 34+0 records in 34+0 records out 17408 bytes (17 kB, 17 KiB) copied, 0.0249334 s, 698 kB/s root@nano:/# gdisk $TARGET_DISK GPT fdisk (gdisk) version 1.0.5 Partition table scan: MBR: not present BSD: not present APM: not present GPT: not present Creating new GPT entries in memory. Command (? for help): n Partition number (1-128, default 1): First sector (34-31277198, default = 2048) or {+-}size{KMGTP}: Last sector (2048-31277198, default = 31277198) or {+-}size{KMGTP}: +10G Current type is 8300 (Linux filesystem) Hex code or GUID (L to show codes, Enter = 8300): 0700 Changed type of partition to 'Microsoft basic data' Command (? for help): n Partition number (2-128, default 2): First sector (34-31277198, default = 20973568) or {+-}size{KMGTP}: Last sector (20973568-31277198, default = 31277198) or {+-}size{KMGTP}: Current type is 8300 (Linux filesystem) Hex code or GUID (L to show codes, Enter = 8300): Changed type of partition to 'Linux filesystem' Command (? for help): p Disk /dev/sda: 31277232 sectors, 14.9 GiB Model: Extreme Sector size (logical/physical): 512/512 bytes Disk identifier (GUID): 0E42CF4F-3EB9-420E-A5A1-6AE9E3D746F0 Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 31277198 Partitions will be aligned on 2048-sector boundaries Total free space is 2014 sectors (1007.0 KiB) Number Start (sector) End (sector) Size Code Name 1 2048 20973567 10.0 GiB 0700 Microsoft basic data 2 20973568 31277198 4.9 GiB 8300 Linux filesystem Command (? for help): w Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING PARTITIONS!! Do you want to proceed? (Y/N): Y OK; writing new GUID partition table (GPT) to /dev/sda. The operation has completed successfully. root@nano:/# mkfs.vfat ${TARGET_DISK}1 mkfs.fat 4.1 (2017-01-24) root@nano:/# mkfs.ext3 -L casper-rw ${TARGET_DISK}2 mke2fs 1.45.6 (20-Mar-2020) Creating filesystem with 1287953 4k blocks and 322560 inodes Filesystem UUID: b8f3305d-17ab-4344-a611-b2817fe8fcb3 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736 Allocating group tables: done Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: done root@nano:/# mount -o loop /mnt/ssd/focal-desktop-amd64.iso /mnt/iso mount: /mnt/iso: WARNING: device write-protected, mounted read-only. root@nano:/# mount ${TARGET_DISK}1 /mnt/hd root@nano:/# cp -rT /mnt/iso /mnt/hd cp: cannot create symbolic link '/mnt/hd/ubuntu': Operation not permitted cp: cannot create symbolic link '/mnt/hd/dists/stable': Operation not permitted cp: cannot create symbolic link '/mnt/hd/dists/unstable': Operation not permitted root@nano:/# ## The following adds the 'persistent' option to 'grub.cfg' root@nano:/# sed -i 's/\/casper\/vmlinuz\$casper_flavour /\/casper\/vmlinuz\$casper_flavour persistent /g' /mnt/hd/boot/grub/grub.cfg root@nano:/# umount /mnt/hd root@nano:/# umount /mnt/iso root@nano:/# sync ----------------------------------------------------------------------------------------- Then, when you boot the media on a UEFI platform, the persistent partition consistently fails to mount (I believe I saw it actually mount twice in about 20 tryouts, which does indeed seem to indicate a race condition). FYI, this is what you may see in Ubuntu when the persistent media above fails to activate persistence: ----------------------------------------------------------------------------------------- To run a command as administrator (user "root"), use "sudo ". See "man sudo_root" for details. ubuntu@ubuntu:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 7.7G 0 7.7G 0% /dev tmpfs 1.6G 1.9M 1.6G 1% /run /dev/sda1 10G 2.7G 7.4G 27% /cdrom /dev/loop0 2.1G 2.1G 0 100% /rofs /cow 7.7G 129M 7.6G 2% / tmpfs 7.7G 0 7.7G 0% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 7.7G 0 7.7G 0% /sys/fs/cgroup tmpfs 7.7G 0 7.7G 0% /tmp tmpfs 1.6G 44K 1.6G 1% /run/user/999 /dev/sda2 4.8G 10M 4.6G 1% /media/ubuntu/casper-rw /dev/loop1 94M 94M 0 100% /snap/core/8935 ubuntu@ubuntu:~$ cat /var/log/casper.log mount: mounting /cow on /root/cow failed: No such file or directory passwd: password expiry information changed. dbus-daemon[1069]: [session uid=999 pid=1069] Activating service name='org.gtk.vfs.Daemon' requested by ':1.0' (uid=999 pid=1070 comm="" label="unconfined") dbus-daemon[1069]: [session uid=999 pid=1069] Successfully activated service 'org.gtk.vfs.Daemon' dbus-daemon[1069]: [session uid=999 pid=1069] Activating service name='org.gtk.vfs.Metadata' requested by ':1.0' (uid=999 pid=1070 comm="" label="unconfined") fuse: device not found, try 'modprobe fuse' first dbus-daemon[1069]: [session uid=999 pid=1069] Successfully activated service 'org.gtk.vfs.Metadata' A connection to the bus can't be made Using CD-ROM mount point /cdrom/ Identifying... [e4f884762ec30c968a13f203cdef4b44-2] Scanning disc for index files... Found 2 package indexes, 0 source indexes, 0 translation indexes and 1 signatures Found label 'Ubuntu 20.04 LTS _Focal Fossa_ - Beta amd64 (20200402)' This disc is called: 'Ubuntu 20.04 LTS _Focal Fossa_ - Beta amd64 (20200402)' Copying package lists...gpgv: Signature made Fri Apr 3 07:35:50 2020 UTC gpgv: using RSA key D94AA3F0EFE21092 gpgv: Good signature from "Ubuntu CD Image Automatic Signing Key (2012)