mprotect fails on ext4 with dax

Bug #1799237 reported by Igor Chorazewicz on 2018-10-22
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Ubuntu
Undecided
Unassigned
linux (Ubuntu)
High
Joseph Salisbury
Bionic
High
Joseph Salisbury
pmdk (Ubuntu)
Undecided
Unassigned

Bug Description

I have a file located on ext4 mounted with "dax". When I call mmap on that file with protection flag different than PROT_NONE and pass the returned address to mprotect(..., PROT_NONE) it fails with:
mprotect: Permission denied

This bug affects PMDK (https://github.com/pmem/pmdk) and seems to be Ubuntu kernel-specific.
Problem was observer on kernel 4.15.0-36-generic and 4.15.0-34-generic

Below is a code which can be used to reproduce the issue.

#include <sys/stat.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>

int main(int argc, char *argv[])
{
               if (argc < 3) {
                              fprintf(stderr, "usage %s file size\n", argv[0]);
                              return 1;
               }

               int size = atoi(argv[2]);

               int fd = open(argv[1], O_RDWR);
               if (fd < 0) {
                              perror("open");
                              return 1;
               }

               void *addr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
               if (addr == MAP_FAILED) {
                              perror("mmap");
                              return 1;
               }

               if(mprotect(addr, size, PROT_NONE)) {
                              perror("mprotect");
                              return 1;
               }

               return 0;
}
---
ProblemType: Bug
ApportVersion: 2.20.9-0ubuntu7.4
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-10-23 (0 days ago)
InstallationMedia: Ubuntu 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180725)
IwConfig:
 lo no wireless extensions.

 enp0s3 no wireless extensions.
Lsusb:
 Bus 001 Device 002: ID 80ee:0021 VirtualBox USB Tablet
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: innotek GmbH VirtualBox
Package: linux (not installed)
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-36-generic root=UUID=48e87c4c-3028-4252-b7bb-e1e6091ff7f6 ro quiet splash
ProcVersionSignature: Ubuntu 4.15.0-36.39-generic 4.15.18
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-36-generic N/A
 linux-backports-modules-4.15.0-36-generic N/A
 linux-firmware 1.173.1
RfKill:

Tags: bionic
Uname: Linux 4.15.0-36-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 12/01/2006
dmi.bios.vendor: innotek GmbH
dmi.bios.version: VirtualBox
dmi.board.name: VirtualBox
dmi.board.vendor: Oracle Corporation
dmi.board.version: 1.2
dmi.chassis.type: 1
dmi.chassis.vendor: Oracle Corporation
dmi.modalias: dmi:bvninnotekGmbH:bvrVirtualBox:bd12/01/2006:svninnotekGmbH:pnVirtualBox:pvr1.2:rvnOracleCorporation:rnVirtualBox:rvr1.2:cvnOracleCorporation:ct1:cvr:
dmi.product.family: Virtual Machine
dmi.product.name: VirtualBox
dmi.product.version: 1.2
dmi.sys.vendor: innotek GmbH

description: updated

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1799237

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: bionic

apport information

tags: added: apport-collected
description: updated

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu:
status: New → Confirmed
dawid (dawpalu) on 2018-10-24
description: updated
Changed in pmdk (Ubuntu):
status: New → Confirmed
Andreas Hasenack (ahasenack) wrote :

Is there something to be done in the user space pmdk component for this bug, or is it purely a kernel issue?

Marcin Ślusarz (mslusarz) wrote :

This is purely kernel issue. It can be worked around in PMDK by code change, as those mprotects are only safety precautions, but I really wouldn't want to do this upstream.

Robie Basak (racb) wrote :

Thanks. If the conclusion is that there's no action needed for pmdk, I think the appropriate status for the pmdk task is Invalid - even though pmdk use is affected. We wouldn't for example add a task for every package if a kernel bug were to cause boot to fail, even though all packages would be affected.

If you want to land a code change in pmdk in Ubuntu though, for example as a workaround, then please reopen the pmdk task.

Changed in pmdk (Ubuntu):
status: Confirmed → Invalid
Changed in ubuntu:
status: Confirmed → Invalid
Joseph Salisbury (jsalisbury) wrote :

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

If it is a regression, we can perform a kernel bisect to narrow down which commit introduced it.

Also, it might be good to test the latest mainline kernel to see if this bug was already fixed upstream. If it was, can investigate to find the commit that fixes the bug.

Changed in linux (Ubuntu):
importance: Undecided → High
no longer affects: pmdk (Ubuntu Bionic)
no longer affects: Ubuntu Bionic
Changed in linux (Ubuntu Bionic):
status: New → Confirmed
importance: Undecided → High
Changed in linux (Ubuntu):
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Bionic):
assignee: nobody → Joseph Salisbury (jsalisbury)
Joseph Salisbury (jsalisbury) wrote :

The latest mainline kernel is available from:
 http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19

Some documentation on it is here:
https://wiki.ubuntu.com/KernelMainlineBuilds

Igor Chorazewicz (igchor) wrote :

This bug was not present in the following kernel: 4.15.0-20-generic

Joseph Salisbury (jsalisbury) wrote :

I'd like to perform a bisect to figure out what commit caused this regression. We need to identify the earliest kernel where the issue started happening as well as the last kernel that did not have this issue.

Can you test the following kernels and report back? Ubuntu 4.15.0-20 was based on the 4.15.17 updates. Ubuntu 4.15.0-23 and newer have the upstream 4.15.18 updates. Testing these two kernels will tell us if the offending commit came in with the 4.15.18 upstream stable updates or if it's specific to Ubuntu:

4.15.17 - http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15.17/
4.15.18 - http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15.18/

Thanks in advance!

Igor Chorazewicz (igchor) wrote :

I have tested the following upstream kernels and mprotect works fine on all of them:
4.15.17-041517-generic
4.15.18-041518-generic
4.19.0-041900-generic

Igor Chorazewicz (igchor) wrote :

It turns out that this issue is only present for certain mapping lengths.
If I run the program, which I attached to the bug report, with size>=2M, mprotect succeeds.

I have also tested this on device dax, and the results depend on alignment.
For 2M alignment, everything works fine, for 4k alignment mprotect fails for all sizes.

Joseph Salisbury (jsalisbury) wrote :

Is this the case with all kernel versions, or can we still consider this a regression in the kernel?

Igor Chorazewicz (igchor) wrote :

Eerything I described in previous comment was observed on Ubuntu kernel (4.15.0-34). It worked fine on upstream and on ubuntu 4.15.0-20 so we can still consider this a regression.

Joseph Salisbury (jsalisbury) wrote :

We can perform a kernel bisect, once we narrow down that last good version and first bad one. Can you test 4.15.0-25? It can be downloaded from:

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/15074499

Igor Chorazewicz (igchor) wrote :

When I install kernel from package I have some problems with pmem emulation and I can't really test this. Here is a simple script which can be used to reproduce the problem (assuming you have pmem emulation, e.g. using memmap and ndctl installed):

sudo umount /dev/pmem0
sudo ndctl create-namespace -f -e namespace0.0 -m fsdax
sudo mkfs.ext4 /dev/pmem0 -F
sudo rm -rf /mnt/pmem
sudo mkdir /mnt/pmem
sudo mount -o dax /dev/pmem0 /mnt/pmem
sudo chmod 777 /mnt/pmem

truncate -s 1M /mnt/pmem/testfile
./test /mnt/pmem/testfile 1048576

'test' is binary of a program presented in the first message.

Marcin Ślusarz (mslusarz) wrote :

To summarize: 4.15.0-20 is the last kernel where mprotect works, all kernels between 4.15.0-20 and 4.15.0-34 are not testable, 4.15.0-34 is the first kernel that boots correctly and mprotect fails.
You have a test program, instructions how to run it and here's a guide how to set up pmem emulation: http://pmem.io/2016/02/22/pm-emulation.html (just adding memmap=1G!4G to kernel command line should be enough).

We (me and Igor) are not kernel developers. The ball is on your side :).

Joseph Salisbury (jsalisbury) wrote :

I started a kernel bisect between Ubuntu 4.15.0-20 and Ubuntu 4.15.0-34. The kernel bisect will require testing of about 7-10 test kernels.

I built the first test kernel, up to the following commit:
9dcfef9fe59a2b4931f58b18fba731d00a4531bd

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1799237

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Igor Chorazewicz (igchor) wrote :

I performed bisect on my own and it seems that following commit causes the problem:
1920d4a1d4ff27ebfd34a99eca67c3c470c8c524 (x86/speculation/l1tf: Invert all not present mappings).

In upstream kernel there is a commit fixing this:
f19f5c49bbc3ffcc9126cc245fc1b24cc29f4a37

The mprotect issue is observed on Ubuntu 4.15.0-36. However, when I apply f19f5c49bbc3ffcc9126cc245fc1b24cc29f4a37 to Ubuntu 4.15.0-36, the issue is gone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers