lxc-sshd won't start with 2.0.8

[Miroslav Los]

On a xenial system after an update to lxc, starting a container created with the lxc-sshd template fails consistently. This does not occur with 2.0.7.

root@xenial:~# lxc-create -n mysshd -t /usr/share/lxc/templates/lxc-sshd
Generating public/private rsa key pair.
Your identification has been saved in /var/lib/lxc/mysshd/rootfs/etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /var/lib/lxc/mysshd/rootfs/etc/ssh/ssh_host_rsa_key.pub.
The key fingerprint is:
SHA256:eR4Kv8JpWxe+RvIudD8LTuOYSGmLdnmX1CgB3Y/IHP4 root@xenial
The key's randomart image is:
+---[RSA 2048]----+
| . . |
| . o . |
| = o o |
| *.. . |
| . So+o |
| ++=Eo. |
| .+++BBo |
| .+B+oO=+o |
| ..o+++== .o |
+----[SHA256]-----+
Generating public/private dsa key pair.
Your identification has been saved in /var/lib/lxc/mysshd/rootfs/etc/ssh/ssh_host_dsa_key.
Your public key has been saved in /var/lib/lxc/mysshd/rootfs/etc/ssh/ssh_host_dsa_key.pub.
The key fingerprint is:
SHA256:Jmet2LLZMtolKBhfDQ/Za4i3yr0/993umj4Hq0D8Qyg root@xenial
The key's randomart image is:
+---[DSA 1024]----+
| |
| o |
| + . |
| . * o o |
|. . + E S o |
| + o + X + . |
|. o o + = o o |
| . + .+B.. ooo. |
| o ++==..oo=*+ |
+----[SHA256]-----+

root@xenial:~# lxc-start -n mysshd --logfile mysshd.log
lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
lxc-start: tools/lxc_start.c: main: 368 To get more details, run the container in foreground mode.
lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.

root@xenial:~# cat mysshd.log
      lxc-start 20170622214710.829 ERROR lxc_conf - conf.c:lxc_setup_dev_console:1473 - Read-only file system - error unlinking /usr/lib/x86_64-linux-gnu/lxc/dev/console
      lxc-start 20170622214710.829 ERROR lxc_conf - conf.c:lxc_setup:4055 - failed to setup the console for 'mysshd'
      lxc-start 20170622214710.829 ERROR lxc_start - start.c:do_start:811 - Failed to setup container "mysshd".
      lxc-start 20170622214710.829 ERROR lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 3)
      lxc-start 20170622214710.868 ERROR lxc_start - start.c:__lxc_start:1358 - Failed to spawn container "mysshd".
      lxc-start 20170622214715.901 ERROR lxc_start_ui - tools/lxc_start.c:main:366 - The container failed to start.
      lxc-start 20170622214715.901 ERROR lxc_start_ui - tools/lxc_start.c:main:368 - To get more details, run the container in foreground mode.
      lxc-start 20170622214715.901 ERROR lxc_start_ui - tools/lxc_start.c:main:370 - Additional information can be obtained by setting the --logfile and --logpriority options.

root@xenial:~# dpkg -l '*lxc*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-======================================-========================-========================-==================================================================================
un liblxc0 <none> <none> (no description available)
ii liblxc1 2.0.8-0ubuntu1~16.04.2 amd64 Linux Containers userspace tools (library)
ii lxc 2.0.8-0ubuntu1~16.04.2 all Transitional package for lxc1
ii lxc-common 2.0.8-0ubuntu1~16.04.2 amd64 Linux Containers userspace tools (common tools)
ii lxc-templates 2.0.8-0ubuntu1~16.04.2 amd64 Linux Containers userspace tools (templates)
ii lxc1 2.0.8-0ubuntu1~16.04.2 amd64 Linux Containers userspace tools
ii lxcfs 2.0.6-0ubuntu1~16.04.1 amd64 FUSE based filesystem for LXC
un lxctl <none> <none> (no description available)
ii python3-lxc 2.0.8-0ubuntu1~16.04.2 amd64 Linux Containers userspace tools (Python 3.x bindings)

Removing the addition of the bind-mount for /dev to the config from the template seems to move the start along, yet implications of doing this are unknown.

[Christian Brauner (cbrauner)]

Hi Miroslav,

Yes, we've been hardening the console handling code quite a bit prior to this release. It seems that you are on a read-only file system which prevents LXC from removing the underlying "/dev/console" file that already exists. LXC wants to remove this file since it wants to prevent bind-mounting over a possible malicious file. Is the read-only filesystem intentional?

[Miroslav Los (mirlos)]

Our actual templates are based on the lxc-sshd template example that comes with lxc-templates. There, basically all the lxc is is bind-mounts for necessary paths from the host, obviously read-only:

# grep mount.entry /usr/share/lxc/templates/lxc-sshd
lxc.mount.entry = /dev dev none ro,bind 0 0
lxc.mount.entry = /lib lib none ro,bind 0 0
lxc.mount.entry = /bin bin none ro,bind 0 0
lxc.mount.entry = /usr usr none ro,bind 0 0
lxc.mount.entry = /sbin sbin none ro,bind 0 0
lxc.mount.entry = tmpfs run/sshd tmpfs mode=0644 0 0
lxc.mount.entry = /usr/share/lxc/templates/lxc-sshd $init_path none ro,bind 0 0
lxc.mount.entry = /etc/init.d etc/init.d none ro,bind 0 0
lxc.mount.entry = /etc/sysconfig/network-scripts etc/sysconfig/network-scripts none ro,bind 0 0
lxc.mount.entry = /etc/rc.d etc/rc.d none ro,bind 0 0
lxc.mount.entry = /lib64 lib64 none ro,bind 0 0

Perhaps bind-mounting /dev isn't needed anymore, though then I'd like to know why the example does that, and what the implications are of leaving the /dev entry out.

[Christian Brauner (cbrauner)]

On Thu, Jun 22, 2017 at 11:11:59PM -0000, Miroslav Los wrote:
> Our actual templates are based on the lxc-sshd template example that
> comes with lxc-templates. There, basically all the lxc is is bind-mounts
> for necessary paths from the host, obviously read-only:

The /dev bind-mount is definitely not needed anymore since LXC will populate dev
internally on its own. So you can remove this from your template and - if you
want - you can send a PR against LXC master to remove this bind-mount from the
template.
We can however, make the code a little smarter in handling the /dev/console
case by making it ignore unlink() failures.

>
> # grep mount.entry /usr/share/lxc/templates/lxc-sshd
> lxc.mount.entry = /dev dev none ro,bind 0 0
> lxc.mount.entry = /lib lib none ro,bind 0 0
> lxc.mount.entry = /bin bin none ro,bind 0 0
> lxc.mount.entry = /usr usr none ro,bind 0 0
> lxc.mount.entry = /sbin sbin none ro,bind 0 0
> lxc.mount.entry = tmpfs run/sshd tmpfs mode=0644 0 0
> lxc.mount.entry = /usr/share/lxc/templates/lxc-sshd $init_path none ro,bind 0 0
> lxc.mount.entry = /etc/init.d etc/init.d none ro,bind 0 0
> lxc.mount.entry = /etc/sysconfig/network-scripts etc/sysconfig/network-scripts none ro,bind 0 0
> lxc.mount.entry = /etc/rc.d etc/rc.d none ro,bind 0 0
> lxc.mount.entry = /lib64 lib64 none ro,bind 0 0
>
>
> Perhaps bind-mounting /dev isn't needed anymore, though then I'd like to know why the example does that, and what the implications are of leaving the /dev entry out.
>
> --
> You received this bug notification because you are a member of Ubuntu
> containers team, which is subscribed to lxc in Ubuntu.
> Matching subscriptions: lxc
> https://bugs.launchpad.net/bugs/1699903
>
> Title:
> lxc-sshd won't start with 2.0.8
>
> Status in lxc package in Ubuntu:
> New
>
> Bug description:
> On a xenial system after an update to lxc, starting a container
> created with the lxc-sshd template fails consistently. This does not
> occur with 2.0.7.
>
> root@xenial:~# lxc-create -n mysshd -t /usr/share/lxc/templates/lxc-sshd
> Generating public/private rsa key pair.
> Your identification has been saved in /var/lib/lxc/mysshd/rootfs/etc/ssh/ssh_host_rsa_key.
> Your public key has been saved in /var/lib/lxc/mysshd/rootfs/etc/ssh/ssh_host_rsa_key.pub.
> The key fingerprint is:
> SHA256:eR4Kv8JpWxe+RvIudD8LTuOYSGmLdnmX1CgB3Y/IHP4 root@xenial
> The key's randomart image is:
> +---[RSA 2048]----+
> | . . |
> | . o . |
> | = o o |
> | *.. . |
> | . So+o |
> | ++=Eo. |
> | .+++BBo |
> | .+B+oO=+o |
> | ..o+++== .o |
> +----[SHA256]-----+
> Generating public/private dsa key pair.
> Your identification has been saved in /var/lib/lxc/mysshd/rootfs/etc/ssh/ssh_host_dsa_key.
> Your public key has been saved in /var/lib/lxc/mysshd/rootfs/etc/ssh/ssh_host_dsa_key.pub.
> The key fingerprint is:
> SHA256:Jmet2LLZMtolKBhfDQ/Za4i3yr0/993umj4Hq0D8Qyg root@xenial
> The key's randomart image is:
> +---[DSA 1024]----+
> | |
> | o |
> | + . |
> | . * o o |
> |. . + E S o ...