kernel 3.16 breaks POSIX extended ACLs over NFS

Bug #1445664 reported by Nelson Elhage
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Expired
Medium
Unassigned

Bug Description

I have a development environment that uses Vagrant with NFS shares to a Trusty development VM from OS X laptops.

After the upgrade to the HWE kernel 3.16, installing Ruby gems into the NFS mount inside the VM failed because of install(1) failing with permission errors. Debugging revealed that it's trying to set POSIX extended ACLs using setxattr(); On 3.13 this operation failed with EOPNOTSUPP (the server does not support extended ACLs), but on 3.16 it fails with EPERM, even if the NFS mount is explicitly mounted with "noacl".

Debugging with ftrace and source diving suggests that kernel 3.16 refactored the POSIX extended ACL code so that before the call even hits the NFS layer, it passes through a generic permission-check layer. It appears that that layer is not aware of NFS UID remapping, and so fails the call regardless of what the server would have done.

A simple test case is `install -c -m 0755 <some file> <some path in an NFS mount>`, which will succeed on 3.13 and fail on 3.16.

The broken system is:

vagrant@packer-vmware-iso:~$ lsb_release -a; uname -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.2 LTS
Release: 14.04
Codename: trusty
Linux packer-vmware-iso 3.16.0-34-generic #47~14.04.1-Ubuntu SMP Fri Apr 10 17:49:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Dec 22 14:39 seq
 crw-rw---- 1 root audio 116, 33 Dec 22 14:39 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.19
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
HibernationDevice: RESUME=/dev/mapper/packer--vmware--iso--vg-swap_1
InstallationDate: Installed on 2015-09-02 (111 days ago)
InstallationMedia: Ubuntu-Server 14.04.1 LTS "Trusty Tahr" - Release amd64 (20140722.3)
IwConfig: Error: [Errno 2] No such file or directory
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: VMware, Inc. VMware Virtual Platform
Package: linux (not installed)
PciMultimedia:

ProcFB: 0 svgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.16.0-57-generic root=/dev/mapper/packer--vmware--iso--vg-root ro quiet
ProcVersionSignature: Ubuntu 3.16.0-57.77~14.04.1-generic 3.16.7-ckt20
RelatedPackageVersions:
 linux-restricted-modules-3.16.0-57-generic N/A
 linux-backports-modules-3.16.0-57-generic N/A
 linux-firmware 1.127.15
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 3.16.0-57-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip docker lpadmin plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 05/20/2014
dmi.bios.vendor: Phoenix Technologies LTD
dmi.bios.version: 6.00
dmi.board.name: 440BX Desktop Reference Platform
dmi.board.vendor: Intel Corporation
dmi.board.version: None
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 1
dmi.chassis.vendor: No Enclosure
dmi.chassis.version: N/A
dmi.modalias: dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd05/20/2014:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A:
dmi.product.name: VMware Virtual Platform
dmi.product.version: None
dmi.sys.vendor: VMware, Inc.

Revision history for this message
Nelson Elhage (nelhage) wrote :

To be clear, I can easily workaround in my environment by not using the HWE kernel (there's no need for it inside a VM), but this is still a regression that may affect other people and presumably future Ubuntu releases.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-lts-utopic (Ubuntu):
status: New → Confirmed
Revision history for this message
Dominic Scheirlinck (dominics) wrote :

Still seeing this at 3.19 on the HWE kernel, via the linux-generic-lts-vivid package.

Using in the same context as nelhage: an NFS mount via Vagrant.

Changed in linux-lts-vivid (Ubuntu):
status: New → Confirmed
Revision history for this message
penalvch (penalvch) wrote :

Nelson Elhage, thank you for reporting this and helping make Ubuntu better.

Please execute the following command only once, as it will automatically gather debugging information, in a terminal:
apport-collect 1445664

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

tags: added: regression-release utopic
no longer affects: linux-lts-vivid (Ubuntu)
affects: linux-lts-utopic (Ubuntu) → linux (Ubuntu)
Changed in linux (Ubuntu):
importance: Undecided → Medium
status: Confirmed → Incomplete
Revision history for this message
Nelson Elhage (nelhage) wrote : BootDmesg.txt

apport information

tags: added: apport-collected trusty
description: updated
Revision history for this message
Nelson Elhage (nelhage) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Nelson Elhage (nelhage) wrote : Lspci.txt

apport information

Revision history for this message
Nelson Elhage (nelhage) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Nelson Elhage (nelhage) wrote : ProcEnviron.txt

apport information

Revision history for this message
Nelson Elhage (nelhage) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Nelson Elhage (nelhage) wrote : ProcModules.txt

apport information

Revision history for this message
Nelson Elhage (nelhage) wrote : UdevDb.txt

apport information

Revision history for this message
Nelson Elhage (nelhage) wrote : UdevLog.txt

apport information

Revision history for this message
Nelson Elhage (nelhage) wrote : WifiSyslog.txt

apport information

Revision history for this message
penalvch (penalvch) wrote :

Nelson Elhage, at your earliest convenience, could you please test the latest upstream kernel available from the very top line at the top of the page from http://kernel.ubuntu.com/~kernel-ppa/mainline/?C=N;O=D (the release names are irrelevant for testing, and please do not test the daily folder)? Install instructions are available at https://wiki.ubuntu.com/Kernel/MainlineBuilds . This will allow additional upstream developers to examine the issue.

If testing on your main install would be inconvenient, one may:
1) Install Ubuntu to a different partition and then test this there.
2) Backup, or clone the primary install.

If the latest kernel did not allow you to test to the issue (ex. you couldn't boot into the OS) please make a comment in your report about this, and continue to test the next most recent kernel version until you can test to the issue. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this issue is fixed in the mainline kernel, please add the following tags by clicking on the yellow circle with a black pencil icon, next to the word Tags, located at the bottom of the report description:
kernel-fixed-upstream
kernel-fixed-upstream-X.Y-rcZ

Where X, and Y are the first two numbers of the kernel version, and Z is the release candidate number if it exists.

If the mainline kernel does not fix the issue, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-X.Y-rcZ

Please note, an error to install the kernel does not fit the criteria of kernel-bug-exists-upstream.

Once testing of the latest upstream kernel is complete, please mark this report's Status as Confirmed. Please let us know your results.

Thank you for your understanding.

Revision history for this message
Nelson Elhage (nelhage) wrote :

Reproduced on

$ uname -a
Linux packer-vmware-iso 4.4.0-040400rc6-generic #201512202030 SMP Mon Dec 21 01:32:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

tags: added: kernel-bug-exists-upstream kernel-bug-exists-upstream-4.4-rc6
Revision history for this message
penalvch (penalvch) wrote :

Nelson Elhage, the next step is to fully commit bisect from kernel 3.13 to 3.16 in order to identify the last good kernel commit, followed immediately by the first bad one. This will allow for a more expedited analysis of the root cause of your issue. Could you please do this following https://wiki.ubuntu.com/Kernel/KernelBisection ?

Please note, finding adjacent kernel versions is not fully commit bisecting.

Thank you for your understanding.

Helpful bug reporting tips:
https://wiki.ubuntu.com/ReportingBugs

tags: added: needs-bisect
Revision history for this message
Nelson Elhage (nelhage) wrote :

I may or may not have time to do the bisect any time soon, but based on some log-diving, I strongly suspect https://github.com/torvalds/linux/commit/013cdf1088d7235da9477a2375654921d9b9ba9f or the related refactors of being the commit that introduced this regression.

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

Changed in linux (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.