OverlayFS with readonly NFS lower layer. Operation not supported for non-root files.

[kvaps]

Step to reproduse:

    - Crete dircetory with other directories and files.
    - Chown some files in this directory, set any non-root permissions.
    - Export nfs share with 'ro' permissions.
    - Mount nfs share.
    - Mount ovrelayfs with nfs as lowerdir
    - Try to read theese non-root files or directories:

Example for overlay mount:

    # ls -l /var/lib/libvirt
    total 16
    drwx--x--x 2 root root 4096 Oct 10 07:33 boot
    drwx--x--x 2 root root 4096 Oct 10 07:33 images
    drwxr-x--- 3 libvirt-qemu kvm 4096 Jan 11 21:49 qemu
    drwx------ 2 root root 4096 Oct 10 07:33 sanlock

    # ls -l /var/lib/libvirt/qemu/
    ls: cannot open directory '/var/lib/libvirt/qemu/': Operation not supported

And for nfs mount:

    # ls -l /rofs/var/lib/libvirt
    total 16
    drwx--x--x 2 root root 4096 Oct 10 07:33 boot
    drwx--x--x 2 root root 4096 Oct 10 07:33 images
    drwxr-x--- 3 libvirt-qemu kvm 4096 Jan 11 21:49 qemu
    drwx------ 2 root root 4096 Oct 10 07:33 sanlock

    # ls -l /rofs/var/lib/libvirt/qemu/
    total 4
    drwxr-xr-x 3 root root 4096 Jan 11 21:49 channel

If I reset permission, it helps:

    # chown libvirt-qemu:kvm /var/lib/libvirt/qemu
    # ls -l /var/lib/libvirt/qemu/
    total 4
    drwxr-xr-x 3 root root 4096 Jan 11 21:49 channel

My mounts (from /proc/mounts)

    192.168.101.61:/data/opt/ltsp/amd64 /rofs nfs ro,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,port=2049,timeo=7,retrans=10,sec=sys,local_lock=all,addr=192.168.101.61 0 0
    overlay / overlay rw,relatime,lowerdir=/rofs,upperdir=/cow/up,workdir=/cow/work 0 0

I tested it nfs3 and nfs4 mount, with this kenels:

    # uname -a
    Linux controller03 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

    # uname -a
    Linux m1c18n1 4.8.0-36-generic #36~16.04.1-Ubuntu SMP Sun Feb 5 09:39:57 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
---
AlsaDevices:
 total 0
 crw-rw----+ 1 root audio 116, 1 Feb 11 2016 seq
 crw-rw----+ 1 root audio 116, 33 Feb 11 2016 timer
AplayDevices: aplay: device_list:268: no soundcards found...
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
ArecordDevices: arecord: device_list:268: no soundcards found...
AudioDevicesInUse: Error: [Errno 2] No such file or directory
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
DistroRelease: Ubuntu 16.04
IwConfig: Error: [Errno 2] No such file or directory
Lsusb:
 Bus 001 Device 002: ID 8087:8000 Intel Corp.
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: HP ProLiant m710p Server Cartridge
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=vmlinuz-4.8.0-36-generic ro initrd=initrd.img-4.8.0-36-generic init=/sbin/init-ltsp forcepae console=tty1 console=ttyS0,9600 root=/dev/nfs boot=nfs nfsroot=/data/opt/ltsp/amd64 BOOTIF=01-94-57-a5-d3-b9-a6
ProcVersionSignature: Ubuntu 4.8.0-36.36~16.04.1-generic 4.8.11
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-4.8.0-36-generic N/A
 linux-backports-modules-4.8.0-36-generic N/A
 linux-firmware 1.157.6
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial
Uname: Linux 4.8.0-36-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 04/06/2016
dmi.bios.vendor: HP
dmi.bios.version: H06
dmi.board.vendor: HP
dmi.chassis.type: 25
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrH06:bd04/06/2016:svnHP:pnProLiantm710pServerCartridge:pvr:rvnHP:rn:rvr:cvnHP:ct25:cvr:
dmi.product.name: ProLiant m710p Server Cartridge
dmi.sys.vendor: HP

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1665921

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[kvaps (kvapss)]

apport information

[kvaps (kvapss)]

apport information

[kvaps (kvapss)]

apport information

[kvaps (kvapss)]

apport information

[kvaps (kvapss)]

apport information

[kvaps (kvapss)]

apport information

[kvaps (kvapss)]

apport information

[kvaps (kvapss)]

apport information

[kvaps (kvapss)]

apport information

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.10 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10

[kvaps (kvapss)]

I confirm,
It seems that is not overlayfs problem.
The problem occurs only with glusterfs backed storage (on nfs server).
I tested this internal gluster nfs server, and ganesha nfs server, nfs-kernel-server is not supported by glusterfs.
With nfs-kernel-server and ext4 no any problems. Even readonly mode.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]