mprotect fails on ext4 with dax

[Igor Chorazewicz]

I have a file located on ext4 mounted with "dax". When I call mmap on that file with protection flag different than PROT_NONE and pass the returned address to mprotect(..., PROT_NONE) it fails with:
mprotect: Permission denied

This bug affects PMDK (https://github.com/pmem/pmdk) and seems to be Ubuntu kernel-specific.
Problem was observer on kernel 4.15.0-36-generic and 4.15.0-34-generic

Below is a code which can be used to reproduce the issue.

#include <sys/stat.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>

int main(int argc, char *argv[])
{
               if (argc < 3) {
                              fprintf(stderr, "usage %s file size\n", argv[0]);
                              return 1;
               }

               int size = atoi(argv[2]);

               int fd = open(argv[1], O_RDWR);
               if (fd < 0) {
                              perror("open");
                              return 1;
               }

               void *addr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
               if (addr == MAP_FAILED) {
                              perror("mmap");
                              return 1;
               }

               if(mprotect(addr, size, PROT_NONE)) {
                              perror("mprotect");
                              return 1;
               }

               return 0;
}
---
ProblemType: Bug
ApportVersion: 2.20.9-0ubuntu7.4
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-10-23 (0 days ago)
InstallationMedia: Ubuntu 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180725)
IwConfig:
 lo no wireless extensions.

 enp0s3 no wireless extensions.
Lsusb:
 Bus 001 Device 002: ID 80ee:0021 VirtualBox USB Tablet
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: innotek GmbH VirtualBox
Package: linux (not installed)
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-36-generic root=UUID=48e87c4c-3028-4252-b7bb-e1e6091ff7f6 ro quiet splash
ProcVersionSignature: Ubuntu 4.15.0-36.39-generic 4.15.18
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-36-generic N/A
 linux-backports-modules-4.15.0-36-generic N/A
 linux-firmware 1.173.1
RfKill:

Tags: bionic
Uname: Linux 4.15.0-36-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 12/01/2006
dmi.bios.vendor: innotek GmbH
dmi.bios.version: VirtualBox
dmi.board.name: VirtualBox
dmi.board.vendor: Oracle Corporation
dmi.board.version: 1.2
dmi.chassis.type: 1
dmi.chassis.vendor: Oracle Corporation
dmi.modalias: dmi:bvninnotekGmbH:bvrVirtualBox:bd12/01/2006:svninnotekGmbH:pnVirtualBox:pvr1.2:rvnOracleCorporation:rnVirtualBox:rvr1.2:cvnOracleCorporation:ct1:cvr:
dmi.product.family: Virtual Machine
dmi.product.name: VirtualBox
dmi.product.version: 1.2
dmi.sys.vendor: innotek GmbH

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1799237

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Igor Chorazewicz (igchor)]

apport information

[Igor Chorazewicz (igchor)]

apport information

[Igor Chorazewicz (igchor)]

apport information

[Igor Chorazewicz (igchor)]

apport information

[Igor Chorazewicz (igchor)]

apport information

[Igor Chorazewicz (igchor)]

apport information

[Igor Chorazewicz (igchor)]

apport information

[Igor Chorazewicz (igchor)]

apport information

[Igor Chorazewicz (igchor)]

apport information

[Igor Chorazewicz (igchor)]

apport information

[Igor Chorazewicz (igchor)]

apport information

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andreas Hasenack (ahasenack)]

Is there something to be done in the user space pmdk component for this bug, or is it purely a kernel issue?

[Marcin Ślusarz (mslusarz)]

This is purely kernel issue. It can be worked around in PMDK by code change, as those mprotects are only safety precautions, but I really wouldn't want to do this upstream.

[Robie Basak (racb)]

Thanks. If the conclusion is that there's no action needed for pmdk, I think the appropriate status for the pmdk task is Invalid - even though pmdk use is affected. We wouldn't for example add a task for every package if a kernel bug were to cause boot to fail, even though all packages would be affected.

If you want to land a code change in pmdk in Ubuntu though, for example as a workaround, then please reopen the pmdk task.

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

If it is a regression, we can perform a kernel bisect to narrow down which commit introduced it.

Also, it might be good to test the latest mainline kernel to see if this bug was already fixed upstream. If it was, can investigate to find the commit that fixes the bug.

[Joseph Salisbury (jsalisbury)]

The latest mainline kernel is available from:
 http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19

Some documentation on it is here:
https://wiki.ubuntu.com/KernelMainlineBuilds

[Igor Chorazewicz (igchor)]

This bug was not present in the following kernel: 4.15.0-20-generic

[Joseph Salisbury (jsalisbury)]

I'd like to perform a bisect to figure out what commit caused this regression. We need to identify the earliest kernel where the issue started happening as well as the last kernel that did not have this issue.

Can you test the following kernels and report back? Ubuntu 4.15.0-20 was based on the 4.15.17 updates. Ubuntu 4.15.0-23 and newer have the upstream 4.15.18 updates. Testing these two kernels will tell us if the offending commit came in with the 4.15.18 upstream stable updates or if it's specific to Ubuntu:

4.15.17 - http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15.17/
4.15.18 - http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15.18/

Thanks in advance!

[Igor Chorazewicz (igchor)]

I have tested the following upstream kernels and mprotect works fine on all of them:
4.15.17-041517-generic
4.15.18-041518-generic
4.19.0-041900-generic

[Igor Chorazewicz (igchor)]

It turns out that this issue is only present for certain mapping lengths.
If I run the program, which I attached to the bug report, with size>=2M, mprotect succeeds.

I have also tested this on device dax, and the results depend on alignment.
For 2M alignment, everything works fine, for 4k alignment mprotect fails for all sizes.

[Joseph Salisbury (jsalisbury)]

Is this the case with all kernel versions, or can we still consider this a regression in the kernel?

[Igor Chorazewicz (igchor)]

Eerything I described in previous comment was observed on Ubuntu kernel (4.15.0-34). It worked fine on upstream and on ubuntu 4.15.0-20 so we can still consider this a regression.

[Joseph Salisbury (jsalisbury)]

We can perform a kernel bisect, once we narrow down that last good version and first bad one. Can you test 4.15.0-25? It can be downloaded from:

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/15074499

[Igor Chorazewicz (igchor)]

When I install kernel from package I have some problems with pmem emulation and I can't really test this. Here is a simple script which can be used to reproduce the problem (assuming you have pmem emulation, e.g. using memmap and ndctl installed):

sudo umount /dev/pmem0
sudo ndctl create-namespace -f -e namespace0.0 -m fsdax
sudo mkfs.ext4 /dev/pmem0 -F
sudo rm -rf /mnt/pmem
sudo mkdir /mnt/pmem
sudo mount -o dax /dev/pmem0 /mnt/pmem
sudo chmod 777 /mnt/pmem

truncate -s 1M /mnt/pmem/testfile
./test /mnt/pmem/testfile 1048576

'test' is binary of a program presented in the first message.

[Marcin Ślusarz (mslusarz)]

To summarize: 4.15.0-20 is the last kernel where mprotect works, all kernels between 4.15.0-20 and 4.15.0-34 are not testable, 4.15.0-34 is the first kernel that boots correctly and mprotect fails.
You have a test program, instructions how to run it and here's a guide how to set up pmem emulation: http://pmem.io/2016/02/22/pm-emulation.html (just adding memmap=1G!4G to kernel command line should be enough).

We (me and Igor) are not kernel developers. The ball is on your side :).

[Joseph Salisbury (jsalisbury)]

I started a kernel bisect between Ubuntu 4.15.0-20 and Ubuntu 4.15.0-34. The kernel bisect will require testing of about 7-10 test kernels.

I built the first test kernel, up to the following commit:
9dcfef9fe59a2b4931f58b18fba731d00a4531bd

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1799237

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Igor Chorazewicz (igchor)]

I performed bisect on my own and it seems that following commit causes the problem:
1920d4a1d4ff27ebfd34a99eca67c3c470c8c524 (x86/speculation/l1tf: Invert all not present mappings).

In upstream kernel there is a commit fixing this:
f19f5c49bbc3ffcc9126cc245fc1b24cc29f4a37

The mprotect issue is observed on Ubuntu 4.15.0-36. However, when I apply f19f5c49bbc3ffcc9126cc245fc1b24cc29f4a37 to Ubuntu 4.15.0-36, the issue is gone.

[Igor Chorazewicz (igchor)]

The problem is still present on kernel 4.15.0-43. Do you have any plans to fix that?

[Christian Priebe (cpriebe)]

I missed this report when I submitted my own report yesterday (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1813179). In my case there is no workaround, so I am hoping this can be backported. It is a one-line fix.

[Kleber Sacilotto de Souza (kleber-souza)]

Hi,

I have followed the steps from comment #27 and I was able to reproduce the issue with bionic kernel 4.15.0-44-generic. I applied the suggested fix (f19f5c49bbc3 x86/speculation/l1tf: Exempt zeroed PTEs from inversion) and confirmed that it fixes the issue.

An SRU request has been sent with the fix and it's on the way of being applied:
https://lists.ubuntu.com/archives/kernel-team/2019-February/098327.html

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Christian Priebe (cpriebe)]

I have not had a chance to try and reproduce the issue with the example mentioned in this bug report but I have been able to verify that the kernel in bionic-proposed fixes the problem I have reported in the following bug report that has been rightfully marked as a duplicate:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1813179

Steps I took to install the bionic-porposed kernel:

sudo apt-get install software-properties-common
sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -sc)-proposed restricted main universe"
sudo apt -y update
sudo apt-get -t bionic-proposed install linux-generic
 # Reboot

After reboot:

cpriebe@x:~$ uname -a
Linux x 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

cpriebe@x:~$ sudo apt-cache policy linux-generic
linux-generic:
  Installed: 4.15.0.46.48
  Candidate: 4.15.0.46.48
  Version table:
 *** 4.15.0.46.48 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

Considering the other bug has been marked as a duplicate I assume it is okay for me to change the tag on this. If this is not okay, please let me know and/or change it back.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.15.0-46.49

---------------
linux (4.15.0-46.49) bionic; urgency=medium

  * linux: 4.15.0-46.49 -proposed tracker (LP: #1814726)

  * mprotect fails on ext4 with dax (LP: #1799237)
    - x86/speculation/l1tf: Exempt zeroed PTEs from inversion

  * kernel BUG at /build/linux-vxxS7y/linux-4.15.0/mm/slub.c:296! (LP: #1812086)
    - iscsi target: fix session creation failure handling
    - scsi: iscsi: target: Set conn->sess to NULL when iscsi_login_set_conn_values
      fails
    - scsi: iscsi: target: Fix conn_ops double free

  * user_copy in user from ubuntu_kernel_selftests failed on KVM kernel
    (LP: #1812198)
    - selftests: user: return Kselftest Skip code for skipped tests
    - selftests: kselftest: change KSFT_SKIP=4 instead of KSFT_PASS
    - selftests: kselftest: Remove outdated comment

  * RTL8822BE WiFi Disabled in Kernel 4.18.0-12 (LP: #1806472)
    - SAUCE: staging: rtlwifi: allow RTLWIFI_DEBUG_ST to be disabled
    - [Config] CONFIG_RTLWIFI_DEBUG_ST=n
    - SAUCE: Add r8822be to signature inclusion list

  * kernel oops in bcache module (LP: #1793901)
    - SAUCE: bcache: never writeback a discard operation

  * CVE-2018-18397
    - userfaultfd: use ENOENT instead of EFAULT if the atomic copy user fails
    - userfaultfd: shmem: allocate anonymous memory for MAP_PRIVATE shmem
    - userfaultfd: shmem/hugetlbfs: only allow to register VM_MAYWRITE vmas
    - userfaultfd: shmem: add i_size checks
    - userfaultfd: shmem: UFFDIO_COPY: set the page dirty if VM_WRITE is not set

  * Ignore "incomplete report" from Elan touchpanels (LP: #1813733)
    - HID: i2c-hid: Ignore input report if there's no data present on Elan
      touchpanels

  * Vsock connect fails with ENODEV for large CID (LP: #1813934)
    - vhost/vsock: fix vhost vsock cid hashing inconsistent

  * SRU: Fix thinkpad 11e 3rd boot hang (LP: #1804604)
    - ACPI / LPSS: Force LPSS quirks on boot

  * Bionic update: upstream stable patchset 2019-01-17 (LP: #1812229)
    - scsi: sd_zbc: Fix variable type and bogus comment
    - KVM/Eventfd: Avoid crash when assign and deassign specific eventfd in
      parallel.
    - x86/apm: Don't access __preempt_count with zeroed fs
    - x86/events/intel/ds: Fix bts_interrupt_threshold alignment
    - x86/MCE: Remove min interval polling limitation
    - fat: fix memory allocation failure handling of match_strdup()
    - ALSA: hda/realtek - Add Panasonic CF-SZ6 headset jack quirk
    - ARCv2: [plat-hsdk]: Save accl reg pair by default
    - ARC: Fix CONFIG_SWAP
    - ARC: configs: Remove CONFIG_INITRAMFS_SOURCE from defconfigs
    - ARC: mm: allow mprotect to make stack mappings executable
    - mm: memcg: fix use after free in mem_cgroup_iter()
    - mm/huge_memory.c: fix data loss when splitting a file pmd
    - cpufreq: intel_pstate: Register when ACPI PCCH is present
    - vfio/pci: Fix potential Spectre v1
    - stop_machine: Disable preemption when waking two stopper threads
    - drm/i915: Fix hotplug irq ack on i965/g4x
    - drm/nouveau: Use drm_connector_list_iter_* for iterating connectors
    - drm/nouveau: Avoid looping through fake MST connectors
    - gen_stats: Fix netl...