lxc-create cannot setgid

[Luke]

LXC cannot download image, seems like a server error:

~# lxc-create -t download -n test
Setting up the GPG keyring
Downloading the image index
ERROR: Failed to download http://images.linuxcontainers.org//meta/1.0/index-user
lxc-create: lxccontainer.c: create_run_template: 1290 container creation template for test failed
lxc-create: tools/lxc_create.c: main: 318 Error creating container test

Trying to download the file with wget gets the file OK with minor complaints:

~# wget -O /dev/null 'http://images.linuxcontainers.org//meta/1.0/index-user'
URL transformed to HTTPS due to an HSTS policy
--2016-12-01 12:36:58-- https://images.linuxcontainers.org//meta/1.0/index-user
Resolving images.linuxcontainers.org (images.linuxcontainers.org)... 91.189.88.37, 91.189.91.21
Connecting to images.linuxcontainers.org (images.linuxcontainers.org)|91.189.88.37|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://uk.images.linuxcontainers.org/meta/1.0/index-user [following]
--2016-12-01 12:36:58-- https://uk.images.linuxcontainers.org/meta/1.0/index-user
Resolving uk.images.linuxcontainers.org (uk.images.linuxcontainers.org)... 91.189.88.37
Connecting to uk.images.linuxcontainers.org (uk.images.linuxcontainers.org)|91.189.88.37|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9102 (8.9K)
Saving to: ‘/dev/null’

Seems like some SSL problem in the lxc-create binary, specifically the HSTS issue mentioned by wget. Maybe a newly introduced HSTS policy breaks the package?

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: lxc 2.0.5-0ubuntu1.2
ProcVersionSignature: Ubuntu 4.8.0-28.30-generic 4.8.6
Uname: Linux 4.8.0-28-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl
ApportVersion: 2.20.3-0ubuntu8
Architecture: amd64
Date: Thu Dec 1 12:28:28 2016
InstallationDate: Installed on 2016-10-14 (47 days ago)
InstallationMedia: Ubuntu-Server 16.10 "Yakkety Yak" - Release amd64 (20161012.1)
PackageArchitecture: all
SourcePackage: lxc
UpgradeStatus: No upgrade log present (probably fresh install)
dnsmasq.conf:
 dhcp-host=vold,10.0.3.10
 dhcp-host=sftp,10.0.3.11

[Luke (lukasz-bednarczyk+launchpad)]

I should add that existing containers start OK.

[Luke (lukasz-bednarczyk+launchpad)]

adding strace for the command:

~# strace lxc-create -t download -n nginx -- --dist ubuntu --release xenial --arch amd64 2>&1 | tee lxc_strace.log

[Christian Brauner (cbrauner)]

lxc-create does not handle any web requests so this cannot be the cause. Upgrading this to a secure connection is also perfectly fine. Is this reliably reproducible still or was this maybe just a temporary server problem?

[Luke (lukasz-bednarczyk+launchpad)]

The issue seems permanent, for the time being.
Running a more thorough strace (attached) has revealed that the download
is indeed handled by the /usr/share/lxc/templates/lxc-download binary,
which unfortunately refuses to work if invoked directly by shell, so
unfortunately I could not debug this particular process any further.

[Luke (lukasz-bednarczyk+launchpad)]

My mistake, actually it is a shell script. Will look into it.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christopher Townsend (townsend)]

I see this today on a Zesty host when trying to create containers. I do not see this issue on a Xenial host however.

[Luke (lukasz-bednarczyk+launchpad)]

I have a suspicion that the error is related to the uid/gid mappings. I need several mappings for different containers. It all starts to creep up on any machine configured like so:

/etc/subuid
------------------------
root:100000:65536
root:33:1
root:100034:65503
root:503:1
root:100504:65033
------------------------

/etc/subgid
------------------------
root:100000:65536
root:33:1
root:100034:65503
root:109:1
root:100110:65427
------------------------

My hunch is that the download script fails to recognize which mapping it should use for the container filesystem it is extracting onto the disk.

[Luke (lukasz-bednarczyk+launchpad)]

Problem occurs even with the secondary mappings in /etc/lxc/default.conf hashed out:

-------------------------------
lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536

#lxc.id_map = u 0 100000 503
#lxc.id_map = u 503 503 1
#lxc.id_map = u 504 100504 65033

#lxc.id_map = g 0 100000 109
#lxc.id_map = g 109 109 1
#lxc.id_map = g 110 100110 65427
-------------------------------

[Christian Brauner (cbrauner)]

Hi,

Have you tried again after a while. I don't think that this is related to the
uid/gid mappings. In order for the download template to work you should have a
default lxc config for your unprivileged user configured which would list the
uid/gid mapping you want to use, e.g.

# Container specific configuration
lxc.id_map = u 0 165536 65536
lxc.id_map = g 0 165536 65536

and that's the mapping lxc would use so it shouldn't get confused by overlapping
mappings for one and the same user. Also, I can't reproduce this by using
overlapping mappings.

Christian

On Thu, Jan 05, 2017 at 10:08:31AM -0000, Luke wrote:
> I have a suspicion that the error is related to the uid/gid mappings. I
> need several mappings for different containers. It all starts to creep
> up on any machine configured like so:
>
> /etc/subuid
> ------------------------
> root:100000:65536
> root:33:1
> root:100034:65503
> root:503:1
> root:100504:65033
> ------------------------
>
> /etc/subgid
> ------------------------
> root:100000:65536
> root:33:1
> root:100034:65503
> root:109:1
> root:100110:65427
> ------------------------
>
> My hunch is that the download script fails to recognize which mapping it
> should use for the container filesystem it is extracting onto the disk.
>
> --
> You received this bug notification because you are a member of Ubuntu
> containers team, which is subscribed to lxc in Ubuntu.
> Matching subscriptions: lxc
> https://bugs.launchpad.net/bugs/1646462
>
> Title:
> lxc container download error (possibly HSTS related)
>
> Status in lxc package in Ubuntu:
> Confirmed
>
> Bug description:
> LXC cannot download image, seems like a server error:
>
> ~# lxc-create -t download -n test
> Setting up the GPG keyring
> Downloading the image index
> ERROR: Failed to download http://images.linuxcontainers.org//meta/1.0/index-user
> lxc-create: lxccontainer.c: create_run_template: 1290 container creation template for test failed
> lxc-create: tools/lxc_create.c: main: 318 Error creating container test
>
> Trying to download the file with wget gets the file OK with minor
> complaints:
>
> ~# wget -O /dev/null 'http://images.linuxcontainers.org//meta/1.0/index-user'
> URL transformed to HTTPS due to an HSTS policy
> --2016-12-01 12:36:58-- https://images.linuxcontainers.org//meta/1.0/index-user
> Resolving images.linuxcontainers.org (images.linuxcontainers.org)... 91.189.88.37, 91.189.91.21
> Connecting to images.linuxcontainers.org (images.linuxcontainers.org)|91.189.88.37|:443... connected.
> HTTP request sent, awaiting response... 301 Moved Permanently
> Location: https://uk.images.linuxcontainers.org/meta/1.0/index-user [following]
> --2016-12-01 12:36:58-- https://uk.images.linuxcontainers.org/meta/1.0/index-user
> Resolving uk.images.linuxcontainers.org (uk.images.linuxcontainers.org)... 91.189.88.37
> Connecting to uk.images.linuxcontainers.org (uk.images.linuxcontainers.org)|91.189.88.37|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 9102 (8.9K)
> Saving to: ‘/dev/null’
>
> Seems like some SSL problem in the lxc-create binary, specifically the
> HSTS issue m...

[Luke (lukasz-bednarczyk+launchpad)]

Problem also occurs with the defaults in /etc/lxc/default.conf.
However, the mappings are defined also in /etc/suguid and /etc/subgid, where the mapping also overlap, like so:

/etc/subuid
----------------------
lxd:100000:65536
root:100000:65536
root:33:1
root:100034:65503
root:503:1
root:100504:65033
----------------------

/etc/subgid
----------------------
lxd:100000:65536
root:100000:65536
root:33:1
root:100034:65503
root:109:1
root:100110:65427
----------------------

I don't think I should reset those to default while some containers are running.

[Ken Sharp (kennybobs)]

I'm pretty certain this is not related to HSTS, and it is a setuid issue:

[pid 19145] openat(3, "uid_map", O_WRONLY|O_LARGEFILE) = 6
[pid 19145] write(6, "0 10000 1\n1001 1001 1\n", 22) = -1 EPERM (Operation not permitted)
[pid 19145] write(2, "newuidmap: write to uid_map fail"..., 60) = 60
[pid 19142] <... read resumed> "newuidmap: write to uid_map fail"..., 4095) = 60
[pid 19145] exit_group(1) = ?
[pid 19142] waitpid(19144, <unfinished ...>
[pid 19145] +++ exited with 1 +++
[pid 19144] <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 19145
[pid 19144] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19145, si_uid=1001, si_status=1, si_utime=0, si_stime=0} ---
[pid 19144] sigreturn({mask=[]}) = 19145
[pid 19144] exit_group(1) = ?
[pid 19144] +++ exited with 1 +++
[pid 19142] <... waitpid resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0) = 19144
[pid 19142] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19144, si_uid=1001, si_status=1, si_utime=0, si_stime=0} ---
[pid 19142] close(5) = 0
[pid 19142] write(2, "lxc 20180712141840.743 ERROR "..., 204 <unfinished ...>
[pid 19141] <... read resumed> "lxc 20180712141840.743 ERROR "..., 4095) = 204
[pid 19141] waitpid(19142, <unfinished ...>
[pid 19142] <... write resumed> ) = 204
[pid 19142] write(2, "error mapping child\n", 20) = 20
[pid 19142] write(7, "1", 1 <unfinished ...>
[pid 19143] <... read resumed> "1", 1) = 1
[pid 19143] close(5) = 0
[pid 19143] close(6) = 0
[pid 19143] setgid32(0) = -1 EINVAL (Invalid argument)
[pid 19143] dup(2) = 4
[pid 19143] fcntl64(4, F_GETFL) = 0x1 (flags O_WRONLY)
[pid 19142] <... write resumed> ) = 1
[pid 19143] close(4 <unfinished ...>
[pid 19142] waitpid(19143, <unfinished ...>
[pid 19143] <... close resumed> ) = 0
[pid 19143] write(2, "setgid: Invalid argument\n", 25) = 25
[pid 19143] write(1, "WARN: could not reopen tty: No s"..., 108) = 108
[pid 19143] exit_group(-1) = ?
[pid 19143] +++ exited with 255 +++
[pid 19142] <... waitpid resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 255}], __WALL) = 19143
[pid 19142] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19143, si_uid=1001, si_status=255, si_utime=0, si_stime=0} ---
[pid 19142] exit_group(255) = ?
[pid 19142] +++ exited with 255 +++
<... waitpid resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 255}], 0) = 19142
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19142, si_uid=1001, si_status=255, si_utime=1, si_stime=2} ---
close(4) = 0
write(2, "Error creating container test\n", 30Error creating container test
) = 30
exit_group(1) = ?
+++ exited with 1 +++

Running lxc-create under sudo -H (I haven't created sub-ids for root) works.

This stops me from creating or running any container, which is wonderful.

[Christian Brauner (cbrauner)]

What's your LXC version?

[Ken Sharp (kennybobs)]

$ lxc-start --version
3.0.1