systemd-nspawn: option --overlay not working

[Olaf Dietsche]

I have a directory with a minimal Xenial installation, which I want to share among several containers. I also have a directory with just Apache and its dependencies installed.

systemd-nspawn has an option --overlay to "Combine multiple directory trees into one overlay file system and mount it into the container."

Neither `systemd-nspawn --overlay=/path/to/xenial:/path/to/apache -D /path/to/container` nor `systemd-nspawn --overlay=/path/to/xenial:/path/to/apache:/path/to/container -D /path/to/container` works as expected.

Both report an error "Directory /path/to/container doesn't look like it has an OS tree. Refusing.". Of course, it doesn't have an OS tree, this is what the base overlays are for.

Looking at the source code, creating a subdirectory /path/to/container/usr works around this. But now it complains about "Failed to create directory /path/to/container/sys/fs/selinux: Read-only file system" twice, then about a missing timezone, and finally aborts with

Creating mount point for overlay /path/to/container/path/to/apache failed: No such file or directory

Doing this manually instead, works fine:

mount -t overlay -o lowerdir=/path/to/apache:/path/to/xenial,upperdir=/path/to/container,workdir=/path/to/workdir none /path/to/container
systemd-nspawn -D /path/to/container

1)
Description: Ubuntu 16.04 LTS
Release: 16.04

2)
systemd-container:
  Installed: 229-4ubuntu6
  Candidate: 229-4ubuntu6
  Version table:
 *** 229-4ubuntu6 500
        500 http://de.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
        100 /var/lib/dpkg/status
     229-4ubuntu4 500
        500 http://de.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages

3)
I expect systemd-nspawn to mount an overlayfs with the provided directories and then start the container with all the directories combined.

4)
systemd-nspawn complains

- Directory /path/to/container doesn't look like it has an OS tree. Refusing.

and after working around this (among other irrelevant error messages)

- Creating mount point for overlay /path/to/container/path/to/apache failed: No such file or directory

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Juanjo Presa (juanjop)]

Neither overlay working here. I think that the root problem is systemd-nspawn never mounts the overlay. Please, can you can check it by defining --overlay and -D to different dirs and then check mounts in host? Is this behaviour correct?

[Juanjo Presa (juanjop)]

Ok, I'm misunderstanding overlay option. I think that you too.

In your example "systemd-nspawn --overlay=/path/to/xenial:/path/to/apache:/path/to/container -D /path/to/container" really the last path of overlay option is path INSIDE container. So systemd-nspawn refuse to init the container because /path/to/container is actually empty.

Anyways there is some request in systemd to support something like --overlay=/path/to/xenial:/path/to/apache:/ and get overlay mounted as rootfs of the container.

I've written some topic in systemd-devel list asking for suggestions:
https://lists.freedesktop.org/archives/systemd-devel/2016-August/037355.html

Also a github issue asking for rootfs support:
https://github.com/systemd/systemd/issues/3847

[Olaf Dietsche (olaf.dietsche)]

Tank you for pointing to the feature request and the discussion.

---

From the man page (https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html#--overlay=):

>For details about overlay file systems, see overlayfs.txt.

Which links to https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt, and there it is clearly defined, how overlayfs works.

So whatever nspawn does, it doesn't seem to conform to Linux kernel's notion of overlay file system.

[Olaf Dietsche (olaf.dietsche)]

It seems, you're right with the final path component in --overlay. This is the absolute path *inside* the container. I looked into nspawn's source code and the options is interpreted as follows

    --overlay=lo1:lo2:up:dest

lo1 and lo2 are lower directories, up is the upper directory in overlayfs terms, and all are absolute host directory paths. dest is the where the overlayfs is mounted inside the container's root directory.

In order to use --overlay, I have to say

    --overlay=/path/to/xenial/usr:/path/to/apache/usr:/path/to/container/usr:/usr

and repeat this for /etc, /bin, /sbin, /lib, /lib64, /var.

So --overlay seems to work, sort of.

Anyway, thank you for pushing me into the right direction. I think, I'll stay with my workaround, overlay mounting the base directories upfront, and then nspawning the container on the resulting root directory.

[Dan Streetman (ddstreet)]

please reopen if this is still an issue