fstrim: cannot open /dev/.lxd-mounts: Permission denied

Bug #1589289 reported by Tamas Papp on 2016-06-05
152
This bug affects 28 people
Affects Status Importance Assigned to Milestone
util-linux (Debian)
Confirmed
Unknown
util-linux (Ubuntu)
Medium
Eric Desrochers
Xenial
Medium
Eric Desrochers
Bionic
Medium
Eric Desrochers
Disco
Medium
Eric Desrochers

Bug Description

[Impact]
fstrim weekly cronjob output in an unprivileged LXD container:

/etc/cron.weekly/fstrim:
fstrim: cannot open /dev/.lxd-mounts: Permission denied
fstrim: /dev/fuse: not a directory
fstrim: /dev/lxd: FITRIM ioctl failed: Operation not permitted

There is a github issue:

https://github.com/lxc/lxd/issues/2030

The outcome is that it's purely an fstrim misbehaviour, it could be smarter.

Stephane Graber comment:

As all of this is handled by the kernel, there isn't anything we can do about it in LXD.

I think fstrim should be made slightly more clever:

* Don't run on bind-mounts (you can detect bind-mounts by parsing /proc/self/mountinfo instead of /proc/mounts)
* Maybe not be as noisy on expected errors like EACCES, EPERM and ENOENT, only log actual failures which would likely be EINVAL or memory related errors.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: util-linux 2.27.1-6ubuntu3
ProcVersionSignature: Ubuntu 4.4.0-21.37-generic 4.4.6
Uname: Linux 4.4.0-21-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
Date: Sun Jun 5 19:49:04 2016
ProcEnviron:
 LANGUAGE=en_US:en
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: util-linux
UpgradeStatus: No upgrade log present (probably fresh install)

[Test Case]
* Ubuntu lxd container
* Wait for the scheduled fstrim run (X: cronjob, B and late: systemd timer)
* fstrim will run and report errors "Operation not permitted" "Permission denied", ...

Container shouldn't run fstrim, it should only be run at host level.

[Potential Regression]

None, the change will only block fstrim to be automatically run at scheduled time. One can still run fstrim on a container manually, even if there is no purpose of doing that.

Xenial uses the cronjob approach /etc/cron.weekly/fstrim
Bionic and late switched to a systemd timer.

2 differents fixes (one for X, and one for B and late) will be needed, but they'll do same thing, which prevent fstrim to automatically run if inside a container both fixes using systemd-virt-detect.

[Other Informations]

* The systemd timer change upstream PR:
https://github.com/karelzak/util-linux/pull/841
https://github.com/karelzak/util-linux/commit/0280d31a2bd6292acd9a4b86d0f6b5feb275a618

CVE References

Tamas Papp (tomposmiko) wrote :
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in util-linux (Ubuntu):
status: New → Confirmed
Changed in util-linux (Ubuntu):
importance: Undecided → Medium
description: updated
Simon Déziel (sdeziel) wrote :

On my containers using the "dir" storage backend, /etc/cron.weekly/fstrim outputs:

fstrim: cannot open /dev/.lxd-mounts: Permission denied
fstrim: /dev/lxd: FITRIM ioctl failed: Operation not permitted
fstrim: /: FITRIM ioctl failed: Operation not permitted

From one of those containers:

# mount | grep sda
/dev/sda on / type ext4 (rw,noatime,errors=remount-ro,data=ordered)
/dev/sda on /dev/lxd type ext4 (rw,noatime,errors=remount-ro,data=ordered)
/dev/sda on /dev/.lxd-mounts type ext4 (rw,noatime,errors=remount-ro,data=ordered)

description: updated
Eric Desrochers (slashd) wrote :

I started to read that bug a couple of days ago

Maybe using 'ConditionVirtualization=!container' should be enought to prevent fstrim from running inside container, I'll give it a try later this week. If its work as expected, I'll submit it upstream.

'container' type would cover the following:

openvz OpenVZ/Virtuozzo
lxc Linux container implementation by LXC
lxc-libvirt Linux container implementation by libvirt
systemd-nspawn systemd's minimal container implementation, see systemd-nspawn(1)
docker Docker container manager
podman Podman container manager
rkt rkt app container runtime
wsl Windows Subsystem for Linux

Reference:
https://www.freedesktop.org/software/systemd/man/systemd-detect-virt.html#

Eric Desrochers (slashd) wrote :

hmmm it is installed under examples/ so not used at all by default.

# dpkg -l util-linux
/usr/share/doc/util-linux/examples/fstrim.service

Eric Desrochers (slashd) wrote :

We could also implement a check mechanism inside "/etc/cron.weekly/fstrim" using'systemd-detect-virt', if it's a container don't run it, otherwise let's do it.

[/etc/cron.weekly/fstrim] # script as-is
#!/bin/sh
# trim all mounted file systems which support it
/sbin/fstrim --all || true

Eric Desrochers (slashd) wrote :

from:
#!/bin/sh
# trim all mounted file systems which support it
/sbin/fstrim --all || true

to:

#!/bin/sh
# trim all mounted file systems which support it
if ! /usr/bin/systemd-detect-virt -q -c; then
/sbin/fstrim --all || true
fi

Eric Desrochers (slashd) wrote :

or even better I think:

#!/bin/sh
# trim all mounted file systems which support it
+if /usr/bin/systemd-detect-virt -q -c; then
+exit 0
+fi
/sbin/fstrim --all || true

Eric Desrochers (slashd) wrote :

Ok more investigation revealed the following

X is using the cron weekly for fstrim, while late version uses fstrim.timer.

So I proposed we fix it with the "ConditionVirtualization=!container" in Bionic and late.

and for X by fixing the cron.weekly/fstrim script that way:

#!/bin/sh
# trim all mounted file systems which support it
+if /usr/bin/systemd-detect-virt -q -c; then
+exit 0
+fi
/sbin/fstrim --all || true

Eric Desrochers (slashd) wrote :

I'll submit the fstrim.timer change upstream.

Eric Desrochers (slashd) on 2019-08-15
Changed in util-linux (Ubuntu):
status: Confirmed → In Progress
assignee: nobody → Eric Desrochers (slashd)
Eric Desrochers (slashd) on 2019-08-20
Changed in util-linux (Ubuntu Xenial):
assignee: nobody → Eric Desrochers (slashd)
status: New → In Progress
Changed in util-linux (Ubuntu Bionic):
assignee: nobody → Eric Desrochers (slashd)
Changed in util-linux (Ubuntu Disco):
assignee: nobody → Eric Desrochers (slashd)
Changed in util-linux (Ubuntu Xenial):
importance: Undecided → Medium
Changed in util-linux (Ubuntu Bionic):
importance: Undecided → Medium
Changed in util-linux (Ubuntu Disco):
importance: Undecided → Medium
Eric Desrochers (slashd) on 2019-08-20
description: updated
tags: added: sts
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in util-linux (Ubuntu Bionic):
status: New → Confirmed
Changed in util-linux (Ubuntu Disco):
status: New → Confirmed
Eric Desrochers (slashd) on 2019-08-20
Changed in util-linux (Ubuntu Bionic):
status: Confirmed → In Progress
Changed in util-linux (Ubuntu Disco):
status: Confirmed → In Progress
description: updated
Eric Desrochers (slashd) wrote :

Upstream PR has been approved/merged.

Eric Desrochers (slashd) wrote :

Uploaded in active devel release (Eoan).

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package util-linux - 2.34-0.1ubuntu2

---------------
util-linux (2.34-0.1ubuntu2) eoan; urgency=medium

  * d/p/prevent-fstrim-inside-container.patch:
    - Prevent fstrim to run inside a container environment.
    (LP: #1589289)

 -- Eric Desrochers <email address hidden> Wed, 21 Aug 2019 13:19:03 +0000

Changed in util-linux (Ubuntu):
status: In Progress → Fix Released

Hello Tamas, or anyone else affected,

Accepted util-linux into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/util-linux/2.33.1-0.1ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in util-linux (Ubuntu Disco):
status: In Progress → Fix Committed
Changed in util-linux (Ubuntu Bionic):
status: In Progress → Fix Committed
Timo Aaltonen (tjaalton) wrote :

Hello Tamas, or anyone else affected,

Accepted util-linux into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/util-linux/2.31.1-0.4ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in util-linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Timo Aaltonen (tjaalton) wrote :

Hello Tamas, or anyone else affected,

Accepted util-linux into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/util-linux/2.27.1-6ubuntu3.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Eric Desrochers (slashd) wrote :

[Bionic verification]

Confirming that I'm using a lxd container running Bionic:

# systemd-detect-virt
lxc

# lsb_release -cs
bionic

* rmadison:
 util-linux | 2.31.1-0.4ubuntu3.3 | bionic-updates
 util-linux | 2.31.1-0.4ubuntu3.4 | bionic-proposed

* With current bionic-updates package:

# dpkg -l | grep -i util-linux
ii util-linux 2.31.1-0.4ubuntu3.3 amd64 miscellaneous system utilities

# systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
   Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Tue 2019-08-27 13:59:23 UTC; 2min 29s ago
  Trigger: Mon 2019-09-02 00:00:00 UTC; 5 days left
     Docs: man:fstrim

Aug 27 13:59:23 lxcfstrim systemd[1]: Started Discard unused blocks once a week.

* With the bionic-proposed package:

# dpkg -l | grep -i util-linux
ii util-linux 2.31.1-0.4ubuntu3.4 amd64 miscellaneous system utilities

# systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
   Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
   Active: inactive (dead) since Tue 2019-08-27 14:02:50 UTC; 23s ago
  Trigger: n/a
Condition: start condition failed at Tue 2019-08-27 14:03:13 UTC; 1s ago
           └─ ConditionVirtualization=!container was not met
     Docs: man:fstrim

Aug 27 13:59:23 lxcfstrim systemd[1]: Started Discard unused blocks once a week.
Aug 27 14:02:50 lxcfstrim systemd[1]: Stopped Discard unused blocks once a week.
Aug 27 14:02:50 lxcfstrim systemd[1]: Stopping Discard unused blocks once a week.

tags: added: verification-done-bionic verification-needed-disco verification-needed-xenial
Eric Desrochers (slashd) wrote :

[Xenial verification]

Confirming that I'm using a lxd container running Xenial:

# systemd-detect-virt
lxc

# lsb_release -cs
xenial

* rmadison:
 util-linux | 2.27.1-6ubuntu3.7 | xenial-updates
 util-linux | 2.27.1-6ubuntu3.8 | xenial-proposed

* With current xenial-updates package:

$ dpkg -l | grep -i util-linux
ii util-linux 2.27.1-6ubuntu3.7 amd64 miscellaneous system utilities

$ sh -xv /etc/cron.weekly/fstrim
#!/bin/sh
# trim all mounted file systems which support it
/sbin/fstrim --all || true
+ /sbin/fstrim --all
fstrim: /: FITRIM ioctl failed: Operation not permitted
+ true

* With current xenial-proposed package:

$ dpkg -l | grep -i util-linux
ii util-linux 2.27.1-6ubuntu3.8 amd64 miscellaneous system utilities

$ sh -xv /etc/cron.weekly/fstrim
#!/bin/sh
# Prevent fstrim to run inside a container environment (LP: #1589289)
if /usr/bin/systemd-detect-virt -q -c; then
exit 0
fi
+ /usr/bin/systemd-detect-virt -q -c
+ exit 0

Eric Desrochers (slashd) wrote :

[Disco verification]

Confirming that I'm using a lxd container running Disco:

# systemd-detect-virt
lxc

# lsb_release -cs
disco

* rmadison:
 util-linux | 2.33.1-0.1ubuntu2 | disco | source, amd64, arm64, armhf, i386, ppc64el, s390x
 util-linux | 2.33.1-0.1ubuntu3 | disco-proposed | source, amd64, arm64, armhf, i386, ppc64el, s390x

* With current disco-updates package:

$ dpkg -l | grep -i util-linux
ii util-linux 2.33.1-0.1ubuntu2 amd64 miscellaneous system utilities

$ systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
   Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Tue 2019-08-27 14:18:00 UTC; 40s ago
  Trigger: Mon 2019-09-02 00:00:00 UTC; 5 days left
     Docs: man:fstrim

Aug 27 14:18:00 lxcdfstrim systemd[1]: Started Discard unused blocks once a week.

* With current disco-proposed package:

$ dpkg -l | grep -i util-linux
ii util-linux 2.33.1-0.1ubuntu3 amd64 miscellaneous system utilities

$ systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
   Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
   Active: inactive (dead) since Tue 2019-08-27 14:19:25 UTC; 43s ago
  Trigger: n/a
Condition: start condition failed at Tue 2019-08-27 14:20:06 UTC; 2s ago
           └─ ConditionVirtualization=!container was not met
     Docs: man:fstrim

Aug 27 14:18:00 lxcdfstrim systemd[1]: Started Discard unused blocks once a week.
Aug 27 14:19:25 lxcdfstrim systemd[1]: fstrim.timer: Succeeded.
Aug 27 14:19:25 lxcdfstrim systemd[1]: Stopped Discard unused blocks once a week.
Aug 27 14:19:25 lxcdfstrim systemd[1]: Stopping Discard unused blocks once a week.
Aug 27 14:19:25 lxcdfstrim systemd[1]: Condition check resulted in Discard unused blocks once a week being skipped.
Aug 27 14:20:06 lxcdfstrim systemd[1]: Condition check resulted in Discard unused blocks once a week being skipped.

tags: added: verification-done-disco verification-done-xenial
removed: verification-needed-disco verification-needed-xenial
description: updated
Eric Desrochers (slashd) wrote :

All regressions for B/D are recurrent patterns that were already failing before that SRU.

For x, there is a chromium-browser failures. It's definitely a chromium-browser thing, not related to the current SRU.

# logs:
Traceback (most recent call last):
  File "/tmp/autopkgtest.Gv56G4/build.V4H/src/debian/tests/chromium-version", line 13, in <module>
    version = driver.capabilities['browserVersion']
KeyError: 'browserVersion'

A new chromium-browser upstream version has been released on "Aug 10 2019":

chromium-browser (76.0.3809.100-0ubuntu0.16.04.1) xenial; urgency=medium

  * Upstream release: 76.0.3809.100
    - CVE-2019-5868: Use-after-free in PDFium ExecuteFieldAction.
    - CVE-2019-5867: Out-of-bounds read in V8.

 -- Olivier Tilloy <email address hidden> Sat, 10 Aug 2019 15:49:36 +0200

Autopkgtest script for that package probably need some rework to adapt to new code reality.

- Eric

All autopkgtests for the newly accepted util-linux (2.31.1-0.4ubuntu3.4) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

network-manager/1.10.6-2ubuntu1.1 (arm64)
openjdk-8/8u222-b10-1ubuntu1~18.04.1 (arm64, ppc64el, armhf, i386, amd64, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#util-linux

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted util-linux (2.33.1-0.1ubuntu3) for disco have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/240-6ubuntu5.3 (amd64, ppc64el)
openjdk-8/8u222-b10-1ubuntu1~19.04.1 (armhf, arm64, amd64, s390x, i386, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/disco/update_excuses.html#util-linux

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted util-linux (2.27.1-6ubuntu3.8) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

chromium-browser/76.0.3809.100-0ubuntu0.16.04.1 (i386, amd64, arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#util-linux

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

The verification of the Stable Release Update for util-linux has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package util-linux - 2.33.1-0.1ubuntu3

---------------
util-linux (2.33.1-0.1ubuntu3) disco; urgency=medium

  * d/p/prevent-fstrim-inside-container.patch:
    - Prevent fstrim to run inside a container environment.
    (LP: #1589289)

 -- Eric Desrochers <email address hidden> Thu, 22 Aug 2019 23:40:49 +0000

Changed in util-linux (Ubuntu Disco):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package util-linux - 2.31.1-0.4ubuntu3.4

---------------
util-linux (2.31.1-0.4ubuntu3.4) bionic; urgency=medium

  * d/p/prevent-fstrim-inside-container.patch:
    - Prevent fstrim to run inside a container environment.
    (LP: #1589289)

 -- Eric Desrochers <email address hidden> Thu, 22 Aug 2019 23:47:46 +0000

Changed in util-linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
Łukasz Zemczak (sil2100) wrote :

As for xenial's chromium-browser test failures - could you please file a bug about the failing architectures? I would then include the bug number in the hint. This way we won't forget about the test issues.

Changed in util-linux (Debian):
status: Unknown → New
Łukasz Zemczak (sil2100) wrote :

Excellent! Let me proceed in that case.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package util-linux - 2.27.1-6ubuntu3.8

---------------
util-linux (2.27.1-6ubuntu3.8) xenial; urgency=medium

  * d/p/prevent-fstrim-inside-container.patch:
    - Prevent fstrim to run inside a container environment.
    (LP: #1589289)

  * d/fstrim-all.cron: Prevent cron.weekly to fstrim if inside a
    container.

 -- Eric Desrochers <email address hidden> Thu, 22 Aug 2019 23:56:21 +0000

Changed in util-linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
Changed in util-linux (Debian):
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.