I/O error while writing superblock on persistent partition with Ubuntu 20.04

[Akeo]

The current (2020.04.07) daily build of Ubuntu 20.04 (focal-desktop-amd64.iso) appears to consistently produce errors on reboot/powerdown when a partition is in use. Occasionally, power down appears to fail after these errors are being displayed altogether.

This was tested on 4 different UEFI machines, with a live persistent media containing the Ubuntu ISO content on a FAT32 first partition, and a 'casper-rw' or 'writable' labelled ext3 second partition (the label used made no difference), and the problem (stream of error messages about trying to access the media beyond the end of the device and inability to write the persistent partition's superblock) manifested itself every single time.

Here's an example of the error messages being displayed from a failed power down, i.e. on a target where the system just freezes during power down, forcing the user to perform a hard reset:

------------------------------------------------------
[ 34.613315] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 34.613332] blk_update_request: I/O error, dev sda, sector 31277056 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 34.613378] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 34.613391] blk_update_request: I/O error, dev sda, sector 31277056 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 34.613411] Buffer I/O error on dev sda, logical block 3909632, async page read
[ 40.209025] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 40.209051] blk_update_request: I/O error, dev sda, sector 18703953 op 0x1:(WRITE) flags 0x800 phys_seg 4 prio class 0
[ 40.209078] Aborting journal on device sda2-8.
[ 40.209099] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 40.209114] blk_update_request: I/O error, dev sda, sector 18695433 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 40.209137] Buffer I/O error on dev sda2, logical block 262, lost sync page write
[ 40.209200] JBD2: Error -5 detected when updating journal superblock for sda2-8.
[ 40.212828] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 40.212851] blk_update_request: I/O error, dev sda, sector 18694337 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 40.212874] Buffer I/O error on dev sda2, logical block 0, lost sync page write
[ 40.212898] EXT4-fs (sda2): I/O error while writing superblock
[ 40.212912] EXT4-fs error (device sda2): ext4_journal_check_start:61: Detected aborted journal
[ 40.212930] EXT4-fs (sda2): remounting file system read-only
[ 40.212946] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 40.212960] blk_update_request: I/O error, dev sda, sector 18694337 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 40.213025] Buffer I/O error on dev sda2, logical block 0, lost sync page write
[ 40.213042] EXT4-fs (sda2): I/O error while writing superblock
------------------------------------------------------

Is it possible that the devices providing access to the bootable media and persistent partition have been deleted or replaced on the system before all accesses have been finished, with the end result being that /dev/sda[#] *regular files* are being used? The errors above would certainly be consistent with /dev/sda[#] no longer mapping to an actual block device, especially as all of the block numbers allegedly residing beyond the end of the device above are inside the highest possible block for the device (31277198 in this case).

To replicate the issue for UEFI boot, you can create a bootable media (e.g. USB Flash Drive) in the following fashion:

------------------------------------------------------
root@nano:/# ## Make sure to change the following disk to your USB media
root@nano:/# export TARGET_DISK=/dev/sda
root@nano:/# ## The following two commands erase the partition tables
root@nano:/# dd if=/dev/zero of=$TARGET_DISK bs=512 count=34
34+0 records in
34+0 records out
17408 bytes (17 kB, 17 KiB) copied, 0.00602524 s, 2.9 MB/s
root@nano:/# dd if=/dev/zero of=$TARGET_DISK bs=512 count=34 seek=$((`blockdev --getsz $TARGET_DISK` - 34))
34+0 records in
34+0 records out
17408 bytes (17 kB, 17 KiB) copied, 0.0249334 s, 698 kB/s
root@nano:/# gdisk $TARGET_DISK
GPT fdisk (gdisk) version 1.0.5

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-31277198, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-31277198, default = 31277198) or {+-}size{KMGTP}: +10G
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 0700
Changed type of partition to 'Microsoft basic data'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-31277198, default = 20973568) or {+-}size{KMGTP}:
Last sector (20973568-31277198, default = 31277198) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'

Command (? for help): p
Disk /dev/sda: 31277232 sectors, 14.9 GiB
Model: Extreme
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 0E42CF4F-3EB9-420E-A5A1-6AE9E3D746F0
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 31277198
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number Start (sector) End (sector) Size Code Name
   1 2048 20973567 10.0 GiB 0700 Microsoft basic data
   2 20973568 31277198 4.9 GiB 8300 Linux filesystem

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): Y
OK; writing new GUID partition table (GPT) to /dev/sda.
The operation has completed successfully.
root@nano:/# mkfs.vfat ${TARGET_DISK}1
mkfs.fat 4.1 (2017-01-24)
root@nano:/# mkfs.ext3 -L casper-rw ${TARGET_DISK}2
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 1287953 4k blocks and 322560 inodes
Filesystem UUID: b8f3305d-17ab-4344-a611-b2817fe8fcb3
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

root@nano:/# mount -o loop /mnt/ssd/focal-desktop-amd64.iso /mnt/iso
mount: /mnt/iso: WARNING: device write-protected, mounted read-only.
root@nano:/# mount ${TARGET_DISK}1 /mnt/hd
root@nano:/# cp -rT /mnt/iso /mnt/hd
cp: cannot create symbolic link '/mnt/hd/ubuntu': Operation not permitted
cp: cannot create symbolic link '/mnt/hd/dists/stable': Operation not permitted
cp: cannot create symbolic link '/mnt/hd/dists/unstable': Operation not permitted
root@nano:/# ## The following adds the 'persistent' option to 'grub.cfg'
root@nano:/# sed -i 's/\/casper\/vmlinuz\$casper_flavour /\/casper\/vmlinuz\$casper_flavour persistent /g' /mnt/hd/boot/grub/grub.cfg
root@nano:/# umount /mnt/hd
root@nano:/# umount /mnt/iso
root@nano:/# sync
------------------------------------------------------

Booting the media above, verifying that the persistent partition is mounted and then shutting down or rebooting the platform should produce the errors.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Iulian Onofrei (revolteh)]

Is there any workaround?