Unable to configure raw.id_map with multiple entries

Bug #1692111 reported by Dawid Wróbel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
lxc (Ubuntu)
Fix Released
Undecided
Christian Brauner
lxd (Ubuntu)
Invalid
Undecided
Christian Brauner

Bug Description

I am trying to map two users (999, 1001) to one of my containers. I added both IDs to /etc/subgid and /etc/subuid. I followed by setting raw.id_map property value (as instructed here https://lists.linuxcontainers.org/pipermail/lxc-users/2017-March/013034.html):

"echo -e "both 999 999\nboth 1001 1001" | lxc config set mycontainer raw.idmap -"

However upon starting the container, I get errors (log excerpt below). If, on the other hand, I set idmap to either "both 999 999" or "both 1001 1001" only - i.e. if I map only one user at the time, the container starts just fine and the user is mapped as expected.

My subgid and subuid look as follows:

lxd:100000:65536
root:100000:65536
root:1001:1
root:999:1

Log excerpt:

Name: mycontainer
Remote: unix:/var/lib/lxd/unix.socket
Architecture: x86_64
Created: 2017/02/22 18:54 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

            lxc 20170519204102.895 ERROR lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/mycontainer" already existed.
            lxc 20170519204102.896 ERROR lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/mycontainer: No such file or directory
            lxc 20170519204102.897 ERROR lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/mycontainer-1" already existed.
            lxc 20170519204102.897 ERROR lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/mycontainer-1: No such file or directory
            lxc 20170519204102.897 ERROR lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/mycontainer-2" already existed.
            lxc 20170519204102.898 ERROR lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/mycontainer-2: No such file or directory
            lxc 20170519204102.898 ERROR lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/mycontainer-3" already existed.
            lxc 20170519204102.898 ERROR lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/mycontainer-3: No such file or directory
            lxc 20170519204102.898 ERROR lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/mycontainer-4" already existed.
            lxc 20170519204102.898 ERROR lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/mycontainer-4: No such file or directory
            lxc 20170519204102.899 ERROR lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/mycontainer-5" already existed.
            lxc 20170519204102.899 ERROR lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/mycontainer-5: No such file or directory
            lxc 20170519204102.899 ERROR lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/mycontainer-6" already existed.
            lxc 20170519204102.899 ERROR lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/mycontainer-6: No such file or directory
            lxc 20170519204102.900 ERROR lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/mycontainer-7" already existed.
            lxc 20170519204102.900 ERROR lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/mycontainer-7: No such file or directory
            lxc 20170519204102.900 ERROR lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/mycontainer-8" already existed.
            lxc 20170519204102.900 ERROR lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/mycontainer-8: No such file or directory
            lxc 20170519204102.901 ERROR lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/mycontainer-9" already existed.
            lxc 20170519204102.901 ERROR lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/mycontainer-9: No such file or directory
            lxc 20170519204102.901 ERROR lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/mycontainer-10" already existed.
            lxc 20170519204102.901 ERROR lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/mycontainer-10: No such file or directory
            lxc 20170519204102.123 ERROR lxc_conf - conf.c:userns_exec_1:4453 - Error setting up child mappings
            lxc 20170519204102.123 ERROR lxc_cgfsng - cgroups/cgfsng.c:cgfsns_chown:1485 - Error requesting cgroup chown in new namespace
            lxc 20170519204102.169 WARN lxc_conf - conf.c:lxc_delete_network:3038 - Failed to remove "vethUVG4PV" from host: Invalid argument.
            lxc 20170519204102.170 ERROR lxc_start - start.c:__lxc_start:1346 - Failed to spawn container "mycontainer".
            lxc 20170519204102.764 ERROR lxc_conf - conf.c:run_buffer:405 - Script exited with status 1.
            lxc 20170519204102.764 ERROR lxc_start - start.c:lxc_fini:546 - Failed to run lxc.hook.post-stop for container "mycontainer".
            lxc 20170519204102.764 WARN lxc_commands - commands.c:lxc_cmd_rsp_recv:172 - Command get_cgroup failed to receive response: Connection reset by peer.
            lxc 20170519204102.764 WARN lxc_commands - commands.c:lxc_cmd_rsp_recv:172 - Command get_cgroup failed to receive response: Connection reset by peer.
            lxc 20170519204102.770 ERROR lxc_conf - conf.c:userns_exec_1:4453 - Error setting up child mappings
            lxc 20170519204102.770 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/systemd//lxc/mycontainer-11
            lxc 20170519204102.775 ERROR lxc_conf - conf.c:userns_exec_1:4453 - Error setting up child mappings
            lxc 20170519204102.775 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/pids//lxc/mycontainer-11
            lxc 20170519204102.781 ERROR lxc_conf - conf.c:userns_exec_1:4453 - Error setting up child mappings
            lxc 20170519204102.781 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/hugetlb//lxc/mycontainer-11
            lxc 20170519204102.787 ERROR lxc_conf - conf.c:userns_exec_1:4453 - Error setting up child mappings
            lxc 20170519204102.787 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/net_cls//lxc/mycontainer-11
            lxc 20170519204102.793 ERROR lxc_conf - conf.c:userns_exec_1:4453 - Error setting up child mappings
            lxc 20170519204102.793 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/cpu//lxc/mycontainer-11
            lxc 20170519204102.798 ERROR lxc_conf - conf.c:userns_exec_1:4453 - Error setting up child mappings
            lxc 20170519204102.798 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/perf_event//lxc/mycontainer-11
            lxc 20170519204102.803 ERROR lxc_conf - conf.c:userns_exec_1:4453 - Error setting up child mappings
            lxc 20170519204102.803 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/memory//lxc/mycontainer-11
            lxc 20170519204102.808 ERROR lxc_conf - conf.c:userns_exec_1:4453 - Error setting up child mappings
            lxc 20170519204102.808 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/freezer//lxc/mycontainer-11
            lxc 20170519204102.813 ERROR lxc_conf - conf.c:userns_exec_1:4453 - Error setting up child mappings
            lxc 20170519204102.813 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/blkio//lxc/mycontainer-11
            lxc 20170519204102.818 ERROR lxc_conf - conf.c:userns_exec_1:4453 - Error setting up child mappings
            lxc 20170519204102.818 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/cpuset//lxc/mycontainer-11
            lxc 20170519204102.823 ERROR lxc_conf - conf.c:userns_exec_1:4453 - Error setting up child mappings
            lxc 20170519204102.823 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/devices//lxc/mycontainer-11

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: lxd 2.0.9-0ubuntu1~16.04.2
ProcVersionSignature: Ubuntu 4.4.0-77.98-generic 4.4.59
Uname: Linux 4.4.0-77-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
Date: Fri May 19 16:42:04 2017
InstallationDate: Installed on 2016-07-28 (295 days ago)
InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719)
ProcEnviron:
 SHELL=/bin/bash
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
SourcePackage: lxd
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Dawid Wróbel (dawidw) wrote :
description: updated
description: updated
description: updated
Revision history for this message
Christian Brauner (cbrauner) wrote :

So the reason why this happens is that internally the id mapping you requested will look like this:

NSID HOSTID RANGE
0 1000000 999
999 999 1
1000 1001000 1
1001 1001 1
1002 1001002 999998998

However, the mappings don't include a mapping for UID 0 with which LXD sets up the container. LXC will internally add the missing mapping for UID 0 which means you are trying to write 6 mappings to the corresponding files in proc. But user namespaces currently enforce a limit of 5 limits.

Changed in lxd (Ubuntu):
status: New → In Progress
Revision history for this message
Christian Brauner (cbrauner) wrote :

This is an interesting problem. So there are two strategies:

- When LXC adds an additional {G,U}ID mapping we could check whether this would lead to more than 5 mappings being requested. And while I sympathize with that I also dislike the fact that the in-kernel limit is arbitrary and I'm not sure if we want to hardcode something that might change at any point. However, we could maybe print the requested mapping count as debug info to the log.
- We find a workaround to only use the mapping for the host {G,U}ID in certain operations where it is required that this {G,U}ID is mapped. I need to think about that.

Revision history for this message
Christian Brauner (cbrauner) wrote :

So I think we might be able to handle this smarter so that users can actually use the full 5 id map ranges. In order to correctly operate on files on the host that we want to chown to the container root user we need only two mappings.
So let's say you want to chown a file to belong to 1000:165536 where the 1000 is your uid on the host and 166536 is the gid of the container root as seen from the host. In order to run a function that executes chown you need:
- a mapping of 65536 (in our example) onto {u,g}id 1000 in a new user namespace
- a mapping of 0 onto 165536 in a new user namespace
Other mappings are not needed. So we can remove the other mappings when executing a function in a new user namespace. The patch for that is somewhat intricate but I'll whip it up soon.

Changed in lxd (Ubuntu):
assignee: nobody → Christian Brauner (cbrauner)
Revision history for this message
Christian Brauner (cbrauner) wrote :

I've got a patch that should improve things so that you can really use five mappings:
https://github.com/lxc/lxc/issues/1598

Revision history for this message
Christian Brauner (cbrauner) wrote :

However, we need to thoroughly review that.

Changed in lxd (Ubuntu):
status: In Progress → Fix Committed
Changed in lxc (Ubuntu):
status: New → Fix Committed
assignee: nobody → Christian Brauner (cbrauner)
Revision history for this message
Stéphane Graber (stgraber) wrote :

Marking the LXD side invalid as this will be sorted out by a liblxc change.

Changed in lxd (Ubuntu):
status: Fix Committed → Invalid
Changed in lxc (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Dawid Wróbel (dawidw) wrote :

Is there any chance this can get backported to Xenial? I can see this fix was included in 2.1.0-0ubuntu1, but the LTS package is still at 2.0.8-0ubuntu1~16.04.2.

Revision history for this message
Stéphane Graber (stgraber) wrote :

I think we have that fix in liblxc 2.0.9 which still needs SRUing to the various Ubuntu stable releases.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.