Inability to use some devices when inside a container

[Miroslav Prašil]

[Impact]

Some /dev/ files such as /dev/null currently cannot be used in nested containers, such as docker in a custom runc container. This is due to the files not being added to the deviceAllowList.

This fix will be included as part of the eventual backport of runc 1.1.5 in supported Ubuntu versions. However, it would be helpful to have sooner for the benefit of users using nested containers.

The issue is fixed by adding a patch containing the upstream commit https://github.com/opencontainers/runc/commit/462e719cae227a990ed793241062a8d2d6145332 which checks /sys/dev for additional /dev files to include.

[Test Plan]

# lxc launch ubuntu:22.04 test-runc -c security.nesting=true -c security.privileged=true
# lxc exec test-runc bash
# apt update && apt dist-upgrade -y
# apt install docker.io runc -y
# docker pull "ubuntu:22.04"
# mkdir -p test-container/rootfs
# cd test-container
# runc spec
> Create a basic linux runc container that has systemd, sh, and docker installed, add the following to the mounts section of config.json file to connect host docker files, and mark readOnly as false for root:

{
    "destination": "/var/run/docker.sock",
    "type": "bind",
    "source": "/var/run/docker.sock",
    "options": [
        "rbind",
        "rw"
    ]
},
{
    "destination": "/var/lib/docker",
    "type": "bind",
    "source": "/var/lib/docker",
    "options": [
        "rbind",
        "rw"
    ]
},

# runc run test-container

/ # cat <<EOF > Dockerfile
FROM ubuntu:22.04

RUN echo test > /dev/null
EOF

/ # docker build -t test .

Before the fix this will result in:
error reopening /dev/null inside container: open /dev/null: operation not permitted: unknown.

This error will no longer happen with the fix in place, and the build will succeed.

[Where problems could occur]

If problems were to occur, they would likely show up in the interaction with the files provided by the /sys directory. If additional files are found, containers could have unintended access to them when they did not before.

[Original Description]

When running nested containers, some devices might not be populated inside the host container. This leads to runc not setting proper `DeviceAllow` options for the container scope which leads to inability to use some devices inside the container. (like /dev/null)

In my specific scenario this led to issues running docker containers on top of system running as LXC container:

https://github.com/opencontainers/runc/discussions/3795

Some more details and fix in runc can be seen here:

https://github.com/opencontainers/runc/pull/3620

This was fixed in runc 1.1.5 that was released yesterday.

My specific system observing this issue is:

# lsb_release -rd
Description: Ubuntu 22.04.2 LTS
Release: 22.04

Although I believe any system using runc 1.1.4 package is affected. My runc version:

# apt-cache policy runc
runc:
  Installed: 1.1.4-0ubuntu1~22.04.1
  Candidate: 1.1.4-0ubuntu1~22.04.1
  Version table:
 *** 1.1.4-0ubuntu1~22.04.1 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.1.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

[Lena Voytek (lvoytek)]

Thank you for the bug report. I created a PPA for 22.04 using the upstream commit you provided here: https://launchpad.net/~lvoytek/+archive/ubuntu/runc-fix-dev-in-containers

If you would like to test it you can run the following commands:

sudo add-apt-repository ppa:lvoytek/runc-fix-dev-in-containers
sudo apt update
sudo apt upgrade

This likely affects kinetic and lunar too, marking as such.

[Miroslav Prašil (cezz)]

Thanks for the packages Lena, I can confirm that 1.1.4-0ubuntu1~22.04.1ppa1 works as expected.

Is there any reason to backport this specific change to 1.1.4 rather than upgrading to 1.1.5?

[Lena Voytek (lvoytek)]

Thank you for the confirmation! I'll get started on adding this fix to Ubuntu.

In general we backport specific changes to the existing releases rather than do a version update in order to avoid unrelated regressions in Ubuntu stable releases. There are exceptions for some packages, but runc is not one of them. If you would like more information on this you can go to https://wiki.ubuntu.com/StableReleaseUpdates

[Miroslav Prašil (cezz)]

Thanks for your work Lena.

As for the update, perhaps I'm misinterpreting the change log, but wasn't runc updated in jammy from 1.1.0 to 1.1.4 about a month ago?

https://launchpad.net/ubuntu/+source/runc/1.1.4-0ubuntu1~22.04.1

I think I'm missing something here. I trust your decision here, but why wasn't the same process followed with 1.1.4 changes? (if that is beyond the work related to this bug, I'm happy to drop it, just curious)

[Lena Voytek (lvoytek)]

Ah, good catch. runc is actually included as part of Docker updates so it will eventually be updated to 1.1.5 or later in stable releases. It may still be useful to add this fix ahead of that depending on when the next backport will be. I will check on that. Thanks!

[Miroslav Prašil (cezz)]

Thanks for explanation. That makes sense to me now.

[Lena Voytek (lvoytek)]

Hi Miroslav,
We decided it'd be useful to add this fix on its own early since the backport of the newer runc version will probably be a few months away. I just uploaded the change to lunar and will get it added to the other releases once it is accepted there.
Thanks for your help!

[Launchpad Janitor (janitor)]

This bug was fixed in the package runc - 1.1.4-0ubuntu3

---------------
runc (1.1.4-0ubuntu3) lunar; urgency=medium

  * d/p/lp2013318-fix-device-files-in-containers.patch: Fix inability to use
    device files such as /dev/null in containers (LP: #2013318)

 -- Lena Voytek <email address hidden> Thu, 06 Apr 2023 10:57:46 -0700

[Steve Langasek (vorlon)]

Hello Miroslav, or anyone else affected,

Accepted runc into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.1.4-0ubuntu1~22.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Miroslav, or anyone else affected,

Accepted runc into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.1.4-0ubuntu1~22.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Miroslav, or anyone else affected,

Accepted runc into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.1.4-0ubuntu1~20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Miroslav Prašil (cezz)]

Hi Steve, I can confirm that runc 1.1.4-0ubuntu1~22.04.2 in ubuntu jammy proposed fixes the issue I'm observing.

The testing I done is following:

1) Downloaded the built package from launchpad:

    # wget 'https://launchpad.net/ubuntu/+source/runc/1.1.4-0ubuntu1~22.04.2/+build/25995926/+files/runc_1.1.4-0ubuntu1~22.04.2_amd64.deb'

2) Installed package with apt:

    # apt install ./runc_1.1.4-0ubuntu1~22.04.2_amd64.deb

3) Verified correct version is indeed installed:

    # dpkg -l runc | grep ii
    ii runc 1.1.4-0ubuntu1~22.04.2 amd64 Open Container Project - runtime

4) Run new docker container:

    # docker run -d alpine sleep 10000
    2f2729419a668b9b2182122147b167caca3c551d2af300ac43d46aab67924020

5) Verified systemd configuration using the returned container ID:

    # systemctl cat docker-2f2729419a668b9b2182122147b167caca3c551d2af300ac43d46aab67924020.scope | grep DeviceAllow
    DeviceAllow=
    DeviceAllow=char-pts rwm
    DeviceAllow=/dev/char/10:200 rwm
    DeviceAllow=/dev/char/5:2 rwm
    DeviceAllow=/dev/char/5:1 rwm
    DeviceAllow=/dev/char/5:0 rwm
    DeviceAllow=/dev/char/1:9 rwm
    DeviceAllow=/dev/char/1:8 rwm
    DeviceAllow=/dev/char/1:7 rwm
    DeviceAllow=/dev/char/1:5 rwm
    DeviceAllow=/dev/char/1:3 rwm
    DeviceAllow=char-* m
    DeviceAllow=block-* m

The above is how I expect the configuration should look like. Broken configuration was missing most of the DevieAllow settings.

6) Ran systemctl daemon-reload and afterwards verified that I still can access /dev/null inside the container as usual:

    # docker exec -ti 2f2729419 cat /dev/null

Prior to fix, exec would fail. And even if I kept shell from docker exec prior to daemon-reload, /dev/null inside the container would be inaccessible.

So as far as I can see, the package runc_1.1.4-0ubuntu1~22.04.2_amd64.deb does fix the issue on jammy.

Hope that helps.

[Lena Voytek (lvoytek)]

Thanks for the verification Miroslav! Confirmed for Kinetic and Focal too:

# lxc launch ubuntu:22.10 test-runc -c security.nesting=true -c security.privileged=true
# lxc exec test-runc bash

# cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

# apt update && apt dist-upgrade -y
# apt install docker.io -y

# docker run -d alpine sleep 1000
366f3646453068167c160e601317ea92e010bf5862bdd91f7637d510de9175ce

# systemctl cat docker-366f3646453068167c160e601317ea92e010bf5862bdd91f7637d510de9175ce.scope | grep DeviceAllow
DeviceAllow=
DeviceAllow=char-pts rwm
DeviceAllow=/dev/char/10:200 rwm
DeviceAllow=/dev/char/5:2 rwm
DeviceAllow=/dev/char/5:1 rwm
DeviceAllow=/dev/char/5:0 rwm
DeviceAllow=/dev/char/1:9 rwm
DeviceAllow=/dev/char/1:8 rwm
DeviceAllow=/dev/char/1:7 rwm
DeviceAllow=/dev/char/1:5 rwm
DeviceAllow=/dev/char/1:3 rwm
DeviceAllow=char-* m
DeviceAllow=block-* m

# docker exec -ti 366f3646453068167 cat /dev/null
> No output

# lxc launch ubuntu:20.04 test-runc -c security.nesting=true -c security.privileged=true
# lxc exec test-runc bash

# cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

# apt update && apt dist-upgrade -y
# apt install docker.io -y

# docker run -d alpine sleep 1000
72cdf46506813662f5da5cba33d26f921f99c09eb3b6f70e5da0945a2943eb9f

# systemctl cat docker-72cdf46506813662f5da5cba33d26f921f99c09eb3b6f70e5da0945a2943eb9f.scope | grep DeviceAllow
DeviceAllow=
DeviceAllow=char-pts rwm
DeviceAllow=/dev/char/10:200 rwm
DeviceAllow=/dev/char/5:2 rwm
DeviceAllow=/dev/char/5:1 rwm
DeviceAllow=/dev/char/5:0 rwm
DeviceAllow=/dev/char/1:9 rwm
DeviceAllow=/dev/char/1:8 rwm
DeviceAllow=/dev/char/1:7 rwm
DeviceAllow=/dev/char/1:5 rwm
DeviceAllow=/dev/char/1:3 rwm
DeviceAllow=char-* m
DeviceAllow=block-* m

# docker exec -ti 72cdf46506813662 cat /dev/null
> No output

[Launchpad Janitor (janitor)]

This bug was fixed in the package runc - 1.1.4-0ubuntu1~22.10.2

---------------
runc (1.1.4-0ubuntu1~22.10.2) kinetic; urgency=medium

  * d/p/lp2013318-fix-device-files-in-containers.patch: Fix inability to use
    device files such as /dev/null in containers (LP: #2013318)

 -- Lena Voytek <email address hidden> Wed, 12 Apr 2023 13:10:10 -0700

[Robie Basak (racb)]

The verification of the Stable Release Update for runc has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package runc - 1.1.4-0ubuntu1~22.04.2

---------------
runc (1.1.4-0ubuntu1~22.04.2) jammy; urgency=medium

  * d/p/lp2013318-fix-device-files-in-containers.patch: Fix inability to use
    device files such as /dev/null in containers (LP: #2013318)

 -- Lena Voytek <email address hidden> Wed, 12 Apr 2023 13:18:53 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package runc - 1.1.4-0ubuntu1~20.04.2

---------------
runc (1.1.4-0ubuntu1~20.04.2) focal; urgency=medium

  * d/p/lp2013318-fix-device-files-in-containers.patch: Fix inability to use
    device files such as /dev/null in containers (LP: #2013318)

 -- Lena Voytek <email address hidden> Wed, 12 Apr 2023 13:21:54 -0700