snapshot persistence broken

[A. Bolmarcich]

I recently tried Ubuntu MATE 15.04 and snapshot persistence does not work. A major problem is that /cow is empty when /sbin/casper-snapshot automatically runs during shutdown. It looks as if the tmpfs that was mounted on /cow during startup has been unmounted before /sbin/casper-snapshot runs.

A minor problem is that the writes to / during the live session end up under /cow/upper but the rest of the snapshot persistence mechanism treats the writes as ending up under /cow (without the subdirectory named upper).

[Martin Wimpress  (flexiondotorg)]

Please can you test this in Ubuntu MATE 15.10?

[A. Bolmarcich (aggedor)]

I just tried Ubunte MATE 15.10-beta2. Snapshot persistence does not work under it for the same reasons that it does not work under 15.04. 15.10-beta2 introduces an additional problem: using a boot argument of "persistent" causes the system to not successfully initialize.

Without a boot argument of "persistent", the "Try Ubuntu MATE without installing" item of the GRUB menu works correctly. The result is a MATE desktop with an open window with the title "Welcome to Ubuntu MATE".

With a boot argument of "persistent", the result is an interactive shell on the console that starts with the lines:

  BusyBox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built-in shell (ash)
  Enter 'help' for a list of built-in commands.

  (initramfs)

At that time, the last few lines of the casper.log file were

  /scripts/casper-bottom/43disable_updateinitramfs: line 44: can't create /root/
usr/sbin/update-initramfs: nonexistent directory
  chmod: /root/usr/sbin/update-initramfs: No such file or directory
  chroot: can't execute 'chown': No such file or directory
  chroot: can't execute 'debconf-copydb': No such file or directory

Entering the command 'exit' in that shell results in lines written to the console which indicate that a panic has occurred.

With boot arguments of "persistent debug" the casper.log file is larger. The contents of that casper.log file indicate that the mountroot() function of /scripts/casper ran at least up to the line preceeding

  kill "$tailpid"

Booting without "persistent" and modifying /etc/casper.conf to make it appear as if /script/casper ran and found a snapshot file, causes /sbin/casper-snapshot to run during shutdown. At that time /cow is an empty directory, as it is with Ubuntu MATE 15.04.

[A. Bolmarcich (aggedor)]

It looks as if I downloaded a Ubuntu MATE 5.10 beta just shortly before the non-beta became available. The non-beta, just as the beta, has a panic in init when booted with an argument of "persistent". What follows are details about the panic in the beta. I have not investigated whether the details are the same in the non-beta, but I suspect they are.

A "No init found" panic occurs in init because an executable sbin/init file is not in the file system built by the mountroot function of /scripts/casper.

A sbin/init file is not there because the mountroot function did not properly do a union mount in the function setup_unionfs of /scripts/casper.

It did not because it did not find any ro filesystems to use in the union mount. It should have found at least /cdrom/casper/filesystem.squashfs. The live boot drive is expected to be mounted at /cdrom at that time.

It did not because earlier in the setup_unionfs function of /scripts/capser, /cdrom was unmounted within the find_cow_device function of /scripts/casper-helpers when that function was used to find a rw persistent filesystem.

The panic does not occur with Ubuntu MATE 15.04, because its setup_unionfs function is not the same as the one in Ubuntu MATE 15.10. In 15.04 the ro filesystems to be used in the union mount are mounted before the find_cow_device function is used. I suspect, but have not tried to confirm, that in 15.04 the unmount of /cdrom done in find_cow_device fails because the device is busy: /cdrom/casper/filesystem.squashfsis is currently mounted.

[A. Bolmarcich (aggedor)]

The "No init found" panic that occurs in init in Ubuntu MATE 15.10 also occurs in the primary distribution of Ubuntu 15.10. It is likely that the panic occurs in all Ubuntu 15.10 based distributions that use the same casper script as the primary distribution.

The panic occurs when using the boot argument of "persistent" and a casper-rw is not found by the time the search for one has searched the boot drive. Previous versions of Ubuntu did not panic under these circumstances. These circumstances are standard ones when using snapshot persistence.

Please fix the bug.

[A. Bolmarcich (aggedor)]

Here is some additional information.

There are critical differences between the filesystems that are present when casper-shutdow automatically runs during shutdown with Ubuntu 15 and with earlier version of Ubuntu. A difference between the two versions of is that Ubuntu 15 uses systemd while the earlier version used init.d files.

With Ubuntu 15.10, the following filesystems that were present during the live session are not present when casper-snapshot automatically runs during shotdown.

where what

/rofs /casper/filesystem.squashfs on boot drive
/cow tmpfs
/tmp tmpfs
/run/user/999 tmpfs
/run/user/999/gvfs gvfsd-fuse
/sys/kernel/debug/tracing tracefs

The first two are present due to the showmounts boot option. /cow is necessary for casper-snapshot to run correctly.

Snapshot presistence works with another Linux distribution that is based on an eariler version of Ubuntu. With that distribution, the following filesystems that were present during the live session are not present when casper-snapshot automatically runs during shotdown.

where what

/tmp tmpfs
/run/user/999/gvfs gvfsd-fuse

In addition, the union filesystem mounted on / that was rw during the live session was ro when casper-snapshot ran during shutdown.

Summary: The systemd equivalent of /etc/init.d/unmountfs is unmounting filesystems needed by casper-snapshot. The systemd equivalent of /etc/init.d/unmountroot is not remounting / as ro.