It's not possible to use OverlayFS (mount -t overlay) to stack directories on a ZFS volume

Bug #1718761 reported by Markus Ueberall
60
This bug affects 12 people
Affects Status Importance Assigned to Milestone
zfs-linux (Ubuntu)
Confirmed
Low
Unassigned

Bug Description

---------- Configuration
# echo -e $(grep VERSION= /etc/os-release)\\nSIGNATURE=\"$(cat /proc/version_signature)\"
VERSION="16.04.3 LTS (Xenial Xerus)"
SIGNATURE="Ubuntu 4.10.0-35.39~16.04.1-generic 4.10.17"
# dpkg --list | grep zfs
ii libzfs2linux 0.6.5.6-0ubuntu18
ii zfs-doc 0.6.5.6-0ubuntu18
ii zfs-initramfs 0.6.5.6-0ubuntu18
ii zfs-zed 0.6.5.6-0ubuntu18
ii zfsutils-linux 0.6.5.6-0ubuntu18

---------- Fault: Creating an overlay of multiple directories on a ZFS volume does not work
# df /var/tmp
Filesystem Type 1K-blocks Used Available Use% Mounted on
tank07/var/tmp zfs 129916288 128 129916160 1% /var/tmp
# mkdir /var/tmp/{lower,middle,upper,workdir,overlay}
# mount -t overlay overlay -olowerdir=/var/tmp/middle:/var/tmp/lower,upperdir=/var/tmp/upper,workdir=/var/tmp/workdir /var/tmp/overlay
mount: wrong fs type, bad option, bad superblock on overlay,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
# dmesg|tail -1
[276328.438284] overlayfs: filesystem on '/var/tmp/upper' not supported as upperdir

---------- Control test 1: Creating an overlay of multiple directories on another filesystem works
# df /tmp
Filesystem Type 1K-blocks Used Available Use% Mounted on
tmpfs tmpfs 1048576 133492 915084 13% /tmp
# mkdir /tmp/{lower,middle,upper,workdir,overlay}
# mount -t overlay overlay -olowerdir=/tmp/middle:/tmp/lower,upperdir=/tmp/upper,workdir=/tmp/workdir /tmp/overlay
# mount | grep overlay
overlay on /tmp/overlay type overlay (rw,relatime,lowerdir=/tmp/middle:/tmp/lower,upperdir=/tmp/upper,workdir=/tmp/workdir)

---------- Control test 2: Creating an overlay using AuFS works on ZFS volume and elsewhere
# mount -t aufs -obr=/tmp/lower:/tmp/middle:/tmp/upper:/tmp/workdir none /tmp/overlay
# mount -t aufs -obr=/var/tmp/lower:/var/tmp/middle:/var/tmp/upper:/var/tmp/workdir none /var/tmp/overlay
# mount | grep aufs
none on /var/tmp/overlay type aufs (rw,relatime,si=9ead1ecae778b250)
none on /tmp/overlay type aufs (rw,relatime,si=9ead1ec9257d1250)

---------- Remark
While the use of AuFS can be used as a workaround in the above scenario, AuFS in turn will not work with [fuse.]glusterfs mounts (this has been documented elsewhere). Given that OverlayFS is part of the (upstream) kernel and Ubuntu now officially supports ZFS, the above should be fixed.
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Sep 18 16:12 seq
 crw-rw---- 1 root audio 116, 33 Sep 18 16:12 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 16.04
IwConfig: Error: [Errno 2] No such file or directory
Lspci: Error: [Errno 2] No such file or directory
Lsusb: Error: [Errno 2] No such file or directory
MachineType: Red Hat KVM
NonfreeKernelModules: zfs zunicode zavl zcommon znvpair
Package: linux (not installed)
PciMultimedia:

ProcFB: 0 cirrusdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/ROOT/ubuntu1604@/boot/vmlinuz-4.10.0-35-generic root=ZFS=tank07/ROOT/ubuntu1604 ro
ProcVersionSignature: Ubuntu 4.10.0-35.39~16.04.1-generic 4.10.17
RelatedPackageVersions:
 linux-restricted-modules-4.10.0-35-generic N/A
 linux-backports-modules-4.10.0-35-generic N/A
 linux-firmware 1.157.12
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial
Uname: Linux 4.10.0-35-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.9.1-5.el7_3.3
dmi.chassis.type: 1
dmi.chassis.vendor: Red Hat
dmi.chassis.version: RHEL 7.3.0 PC (i440FX + PIIX, 1996)
dmi.modalias: dmi:bvnSeaBIOS:bvr1.9.1-5.el7_3.3:bd04/01/2014:svnRedHat:pnKVM:pvrRHEL7.3.0PC(i440FX+PIIX,1996):cvnRedHat:ct1:cvrRHEL7.3.0PC(i440FX+PIIX,1996):
dmi.product.name: KVM
dmi.product.version: RHEL 7.3.0 PC (i440FX + PIIX, 1996)
dmi.sys.vendor: Red Hat
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Sep 18 16:12 seq
 crw-rw---- 1 root audio 116, 33 Sep 18 16:12 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 16.04
IwConfig: Error: [Errno 2] No such file or directory
Lspci: Error: [Errno 2] No such file or directory
Lsusb: Error: [Errno 2] No such file or directory
MachineType: Red Hat KVM
NonfreeKernelModules: zfs zunicode zavl zcommon znvpair
Package: linux (not installed)
PciMultimedia:

ProcFB: 0 cirrusdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/ROOT/ubuntu1604@/boot/vmlinuz-4.10.0-35-generic root=ZFS=tank07/ROOT/ubuntu1604 ro
ProcVersionSignature: Ubuntu 4.10.0-35.39~16.04.1-generic 4.10.17
RelatedPackageVersions:
 linux-restricted-modules-4.10.0-35-generic N/A
 linux-backports-modules-4.10.0-35-generic N/A
 linux-firmware 1.157.12
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial
Uname: Linux 4.10.0-35-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.9.1-5.el7_3.3
dmi.chassis.type: 1
dmi.chassis.vendor: Red Hat
dmi.chassis.version: RHEL 7.3.0 PC (i440FX + PIIX, 1996)
dmi.modalias: dmi:bvnSeaBIOS:bvr1.9.1-5.el7_3.3:bd04/01/2014:svnRedHat:pnKVM:pvrRHEL7.3.0PC(i440FX+PIIX,1996):cvnRedHat:ct1:cvrRHEL7.3.0PC(i440FX+PIIX,1996):
dmi.product.name: KVM
dmi.product.version: RHEL 7.3.0 PC (i440FX + PIIX, 1996)
dmi.sys.vendor: Red Hat

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1718761

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Markus Ueberall (ueberall) wrote : CRDA.txt

apport information

tags: added: apport-collected xenial
description: updated
Revision history for this message
Markus Ueberall (ueberall) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : JournalErrors.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : ProcEnviron.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : ProcModules.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : UdevDb.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : WifiSyslog.txt

apport information

description: updated
Revision history for this message
Markus Ueberall (ueberall) wrote : CRDA.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : JournalErrors.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : ProcEnviron.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : ProcModules.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : UdevDb.txt

apport information

Revision history for this message
Markus Ueberall (ueberall) wrote : WifiSyslog.txt

apport information

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
affects: linux (Ubuntu) → zfs-linux (Ubuntu)
Changed in zfs-linux (Ubuntu):
status: Confirmed → New
Changed in zfs-linux (Ubuntu):
importance: Undecided → Low
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in zfs-linux (Ubuntu):
status: New → Confirmed
Revision history for this message
Scott Moser (smoser) wrote :

I hit this today, and filed bug 1873917 before dupe'ing it to this.
I'd think that the main pain points are
a.) using overlay in a container that is backed by zfs via lxd
b.) using overlay when using zfs root (https://didrocks.fr/2019/10/11/ubuntu-zfs-support-in-19.10-zfs-on-root/)

I attached a re-create https://launchpadlibrarian.net/475448177/test-overlay

Revision history for this message
Nick Niehoff (nniehoff) wrote :

I ran into this exactly as smoser described using using overlay in a container that is backed by zfs via lxd. I believe this will become more prevalent as people start to use zfs root with Focal 20.04.

Revision history for this message
Shane (whitebread) wrote :

I'm using Focal with ZFS on root. I installed the docker snap package today and could not get the service to start. Logs show:

2020-07-04T17:15:20Z docker.dockerd[8105]: time="2020-07-04T12:15:20.381289131-05:00" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2
2020-07-04T17:15:21Z docker.dockerd[8105]: failed to start daemon: error initializing graphdriver: driver not supported
2020-07-04T17:15:21Z systemd[1]: snap.docker.dockerd.service: Main process exited, code=exited, status=1/FAILURE

I think this is the same issue except with docker instead of LXC.

Revision history for this message
Richard Laager (rlaager) wrote :
Revision history for this message
stellarpower (stellarpower) wrote :

I am arriving here as I would also like to run containers (docker) over a zpool as the storage backend. In my case, I am running Ubuntu Server (Focal) on a Pi 3B, I would ideally have root on zfs but with it being a Pi, I have not done this yet. I have an externally attached USB pool for real data and have left the OS as it is on the SD card, however, as the above says, I would like to move more of my systems towards ZFS with time and use Ubuntu-based distros and containers on a lot of them, where these can't be run under illumos as VMs.

I have installed docker-rootless on this system, however due to kernel limitations, it is impossible to mount datasets as non-root on linux (long-standing issue, I am working on a bug report with details of options for a workaround and will attach), and using a noverlay over the top of zfs would be fine for me, as I can snapshot and work with the higher-level system and back up easily, even if not as fast as native zfs performance, however, the daemon will not start, and I have fallen back to the "vfs" driver on my data-root filesystem.

Some googling hasn't indicated what feature(s) would be required in zfs for overlay to be used on top, docker's documentation doesn't seem to clarify anything, and searching for the "fstype=1" requirement in xfs, I find from the manual page for `mkfs.xfs`:
```
ftype=value
                          This feature allows the inode type to be stored in
                          the directory structure so that the readdir(3) and
                          getdents(2) do not need to look up the inode to
                          determine the inode type.
```
which perhaps is not the issue at hand.

Revision history for this message
Colin Ian King (colin-king) wrote :

The proposed fixes are still Work-In-Progress, hopefully they will get merged into ZFS in the next few months.

Revision history for this message
Timo Aaltonen (tjaalton) wrote :

bah, still no fix over a year later

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers