[debian] MountFlags=shared fail for docker 18.09

[Kevin Zhao]

docker version beyond 18.09 now containerd is managed by systemd, Since containerd is managed by systemd under v18.09.1 release, any custom configuration to the docker.service systemd configuration which changes mount settings (for example, MountFlags=slave) breaks interactions between the Docker Engine daemon and containerd, and you will not be able to start containers. Run the following command to get the current value of the MountFlags property for the docker.service.

All the container could not start if we set any "MountFlags" in drop-in file.

[Radosław Piliszek (yoctozepto)]

What distro is it? CentOS 7 is not affected. Both docker and containerd default to MountFlags=shared

[Kevin Zhao (kevin-zhao)]

@Radoslaw,

Centos7 is OK because the default docker version is prior 18.09.

I use Debian stretch, the docker version is 18.09.6

Every docker version lower than 18.09 is OK, but newer than 18.09.0 will cause this issue.

[Radosław Piliszek (yoctozepto)]

@Kevin:
$ rpm -qi docker-ce
Version : 18.09.6
We install upstream version so it is 18.09.6 actually even for CentOS.
We regularly test Ubuntu and CentOS and they do not suffer from this problem so it must be Debian-only.
I will change the bug topic.

[Radosław Piliszek (yoctozepto)]

CentOS docker-ce and containerd.io:
$ rpm -qi containerd.io
Version : 1.2.5
both do not override the default of MountFlags, which is shared, so our drop-in is useless there anyway.

[Kevin Zhao (kevin-zhao)]

@radoslaw,

Thanks for testing. Could you paste the result of:

systemctl show docker.service | grep MountFlags

do you make sure the output has mountFlags? If you specify the mountFlags=shared in kolla.conf, that will induce the error.
===============================================================================================
linaro@j12-m1-c18n1:~$ cat /etc/systemd/system/docker.service.d/kolla.conf
[Service]
MountFlags=shared
ExecStart=
ExecStart=/usr/bin/dockerd --insecure-registry registry.hub.docker.com --log-opt max-file=5 --log-opt max-size=50m
=================================================================================================
linaro@j12-m1-c18n1:~$ sudo docker run -it ubuntu bash
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"bash\": executable file not found in $PATH": unknown.
===============================================================================================

Besides, if you not specify the mountFlags, just delete the line "mountFlags=shared", every thing is OK.

[Kevin Zhao (kevin-zhao)]

btw, if we don't specify the mountFlags=shared in dropped-in conf, the default setting for docker mountFlags is shared?

[Radosław Piliszek (yoctozepto)]

$ systemctl show docker.service | grep MountFlags
MountFlags=1048576
$ systemctl show containerd.service | grep MountFlags
MountFlags=0

Though `man systemd.exec` says:
MountFlags=
           Takes a mount propagation flag: shared, slave or private, which control whether mounts in the file system namespace set up for this unit's processes will receive or propagate mounts or unmounts.
           See mount(2) for details. Defaults to shared.
           ...
           Note that the file system namespace related options
           (PrivateTmp=, PrivateDevices=, ProtectSystem=, ProtectHome=, ReadOnlyDirectories=, InaccessibleDirectories= and ReadWriteDirectories=) require that mount and unmount propagation from the unit's
           file system namespace is disabled, and hence downgrade shared to slave.

None of these are used here.

I only know MountFlags were added to satisfy neutron components which probably had some issues without it.

I am also concerned with the fact that we override the ExecStart line:
$ cat /usr/lib/systemd/system/docker.service | grep ExecStart
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
$ cat /etc/systemd/system/docker.service.d/kolla.conf | grep ExecStart
ExecStart=
ExecStart=/usr/bin/dockerd --log-opt max-file=5 --log-opt max-size=50m

[Radosław Piliszek (yoctozepto)]

We should be using /etc/docker/daemon.json instead anyway:
https://docs.docker.com/engine/reference/commandline/dockerd//#daemon-configuration-file

[Radosław Piliszek (yoctozepto)]

I could not reproduce on clean Debian Stretch updated to the latest package versions and upstream Docker as installed by kolla-ansible bootstrap-servers.
The behavior is the same as for CentOS 7.

[Kevin Zhao (kevin-zhao)]

Hi Radoslaw,
Thanks for helping!

what is the systemd version do you use?

That is related with systemd.

[Kevin Zhao (kevin-zhao)]

$ cat /usr/lib/systemd/system/docker.service | grep ExecStart
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
$ cat /etc/systemd/system/docker.service.d/kolla.conf | grep ExecStart
ExecStart=
ExecStart=/usr/bin/dockerd --log-opt max-file=5 --log-opt max-size=50m

for this, the issue is induced by "mountFlags=shared" in kolla.conf. Do you ever seen it in kolla.conf?

[Mark Goddard (mgoddard)]

Kevin, which OS and version are you using? Docker CE is only supported in kolla-ansible from rocky onwards.

[Kevin Zhao (kevin-zhao)]

@Mark，
Hi, I use debian stretch on Arm64. And docker version 18.09. Kolla-ansile use the lastest commit

image is Rocky.

[Marcin Juszkiewicz (hrw)]

Kevin: have you considered migration to Debian 'buster' and then use 'docker.io' package?

I may dig for machine running Debian 'stretch' but that may need to wait after my PTO ;(

[Mark Goddard (mgoddard)]

Does it just require MountFlags in a drop-in file for containerd?

[Kevin Zhao (kevin-zhao)]

@Marcin

yes, downgrade to docker.io(which will offer prior version to 18.09) will work find. It's easy to go ahead.

@Mark,

If we leave MountFlags=shared in drop-in file, container will not start...

Remove this and restart docker.service, docker works..

[Radosław Piliszek (yoctozepto)]

Kevin,

the only difference between our deployments described so far is the architecture. I don't currently have access to any aarch64/arm64 platform to debug.

May you run there:
systemctl show docker.service | grep MountFlags
systemctl show containerd.service | grep MountFlags
and post the output (just curious if it differs there).

[Kevin Zhao (kevin-zhao)]

@Radoslaw,
Thanks!

sudo systemctl show docker.service | grep MountFlags
MountFlags=shared

sudo systemctl show containerd.service | grep MountFlags
MountFlags=

sudo docker run -it ubuntu bash
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"bash\": executable file not found in $PATH": unknown.

[Radosław Piliszek (yoctozepto)]

Kevin,

odd that they are strings in there, not numerics.

Let's compare also:
$ dpkg-query --show systemd
systemd 232-25+deb9u11

[Mark Goddard (mgoddard)]

Kevin, I mean adding a drop-in file for containerd, as well as Docker.

[Kevin Zhao (kevin-zhao)]

@Mark,
Aha, I've tried and that could not work..Containerd doesn't support the option method just like docker. Will ask containerd community

[Marcin Juszkiewicz (hrw)]

According to Docker upstream release notes [1] MountFlags should be empty.

1. https://docs.docker.com/engine/release-notes/#18091

"Important notes about this release

In Docker versions prior to 18.09, containerd was managed by the Docker engine daemon. In Docker Engine 18.09, containerd is managed by systemd. Since containerd is managed by systemd, any custom configuration to the docker.service systemd configuration which changes mount settings (for example, MountFlags=slave) breaks interactions between the Docker Engine daemon and containerd, and you will not be able to start containers.

Run the following command to get the current value of the MountFlags property for the docker.service:

sudo systemctl show --property=MountFlags docker.service
MountFlags=

Update your configuration if this command prints a non-empty value for MountFlags, and restart the docker service."

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/674587

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/674587
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=35941738d59002f8bbf55d25b3468619983dcea6
Submitter: Zuul
Branch: master

commit 35941738d59002f8bbf55d25b3468619983dcea6
Author: Marcin Juszkiewicz <email address hidden>
Date: Mon Aug 5 14:18:06 2019 +0200

    Stop using MountFlags=shared in Docker configuration

    According to Docker upstream release notes [1] MountFlags should be
    empty.

    1. https://docs.docker.com/engine/release-notes/#18091

    "Important notes about this release

    In Docker versions prior to 18.09, containerd was managed by the Docker
    engine daemon. In Docker Engine 18.09, containerd is managed by systemd.
    Since containerd is managed by systemd, any custom configuration to the
    docker.service systemd configuration which changes mount settings (for
    example, MountFlags=slave) breaks interactions between the Docker Engine
    daemon and containerd, and you will not be able to start containers.

    Run the following command to get the current value of the MountFlags
    property for the docker.service:

    sudo systemctl show --property=MountFlags docker.service
    MountFlags=

    Update your configuration if this command prints a non-empty value for
    MountFlags, and restart the docker service."

    Closes-bug: #1833835

    Change-Id: I4f4cbb09df752d00073a606463c62f0a6ca6c067

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/675299

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/675299
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=5314581beb503f41b360c17678e81a4a7f5f222a
Submitter: Zuul
Branch: stable/stein

commit 5314581beb503f41b360c17678e81a4a7f5f222a
Author: Marcin Juszkiewicz <email address hidden>
Date: Mon Aug 5 14:18:06 2019 +0200

    Stop using MountFlags=shared in Docker configuration

    According to Docker upstream release notes [1] MountFlags should be
    empty.

    1. https://docs.docker.com/engine/release-notes/#18091

    "Important notes about this release

    In Docker versions prior to 18.09, containerd was managed by the Docker
    engine daemon. In Docker Engine 18.09, containerd is managed by systemd.
    Since containerd is managed by systemd, any custom configuration to the
    docker.service systemd configuration which changes mount settings (for
    example, MountFlags=slave) breaks interactions between the Docker Engine
    daemon and containerd, and you will not be able to start containers.

    Run the following command to get the current value of the MountFlags
    property for the docker.service:

    sudo systemctl show --property=MountFlags docker.service
    MountFlags=

    Update your configuration if this command prints a non-empty value for
    MountFlags, and restart the docker service."

    Closes-bug: #1833835

    Change-Id: I4f4cbb09df752d00073a606463c62f0a6ca6c067
    (cherry picked from commit 35941738d59002f8bbf55d25b3468619983dcea6)

[Harri-afaics (harri-afaics)]

Apparently this bug has been fixed for systemd by upstream, see https://github.com/systemd/systemd/commit/37ed15d7edaf59a1fc7c9e3552cd93a83f3814ef

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/745768