[zun] Can't specify Docker runtime when creating a container

[Tomas Stodulka]

Detailed bug description:
 When I specify Docker runtime in creating container via Zun (doesn't matter if it is kata-runtime, or runc/ using Horizon or CLI), status of container ends with error.

Steps to reproduce:
 Create container with choose name, runtime and image, like e.g.:
 zun run --name test-runc --runtime runc cirros

Expected results:
 Create container in status running with choosen runtime.

Actual result:
 Created container is in state error and following error occur in zun-compute:
2019-12-30 12:43:50.084 6 ERROR zun.compute.manager [req-ed76e156-d32e-455a-aabb-9af4da98ceea 5cf1c77016144691ba18ac3a7bef3dfa 26ec8da81d334ed0accd7652301ba13b default - -] Unexpected exception: Specifying runtime in Docker API is not supported: ZunException: Specifying runtime in Docker API is not supported
2019-12-30 12:43:50.084 6 ERROR zun.compute.manager Traceback (most recent call last):
2019-12-30 12:43:50.084 6 ERROR zun.compute.manager File "/var/lib/kolla/venv/lib/python2.7/site-packages/zun/compute/manager.py", line 337, in _do_container_create_base
2019-12-30 12:43:50.084 6 ERROR zun.compute.manager requested_volumes)
2019-12-30 12:43:50.084 6 ERROR zun.compute.manager File "/var/lib/kolla/venv/lib/python2.7/site-packages/zun/container/docker/driver.py", line 277, in create
2019-12-30 12:43:50.084 6 ERROR zun.compute.manager 'Specifying runtime in Docker API is not supported'))
2019-12-30 12:43:50.084 6 ERROR zun.compute.manager ZunException: Specifying runtime in Docker API is not supported

Workaround:
 You can create container via Zun only without specifying runtime. Creating kata or runc containers are functional only directly in Docker app.

Impact:
 We need create environments for testing cyber attacks, easy to deploy with less consumption then VMs. For this purposes we need to use runtimes other then "runc".

Description of the environment:
 Operation system: CentOS Linux release 7.6.1810 (Core)
 Versions of components: Train
 Reference architecture: all-in-one
 Network model: Neutron + Kuryr + vxlan
 Related projects installed: Zun

# openstack --version
 openstack 4.0.0
# docker --version
 Docker version 19.03.5, build 633a0ea
# docker info | grep Runtime
 Runtimes: kata-runtime runc
 Default Runtime: runc

[Radosław Piliszek (yoctozepto)]

This is a limitation of Zun I presume, notifying Zun.

[Tomas Stodulka (xstodu07)]

But where can be the cause? Creating container with choosing Runtime work properly:

[root@controller2 ~]# docker run -dit --runtime kata-runtime cirros
e4347714fca0899fa6376cc51597c26449003e3725f1207521630685f38fcba9
[root@controller2 ~]#

[Radosław Piliszek (yoctozepto)]

Quick search reveals: https://opendev.org/openstack/zun/src/branch/master/zun/container/docker/driver.py#L277
and https://opendev.org/openstack/zun/src/branch/master/zun/container/docker/driver.py#L348
so bumping docker.docker_remote_api_version should help, e.g.

  [docker]
  docker_remote_api_version = 1.40

Possibly in /etc/kolla/zun/zun-compute.conf
and redeploy.

The issue is due to kolla-ansible setting it to a too low value:
docker_remote_api_version = 1.24

I'll mark it on k-a back.

[Radosław Piliszek (yoctozepto)]

@hongbin (Zun PTL) - is there a chance Zun could just discover this instead? (thinking forward)

k-a is safe to remove this as it defaults to 1.26 already

[Tomas Stodulka (xstodu07)]

That's it! Change as you suggested in /etc/kolla/zun-compute/zun.conf and after restart zun_compute container
it is functional. Thank you.

[Radosław Piliszek (yoctozepto)]

I'm glad!

@hongbin: I noticed this is only ever used to check for the runtime availability - maybe due to this and the fact that the threshold version (1.26) is very old, the best option for Zun would be to deprecate and drop this option?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/702034

[hongbin (hongbin034)]

@Radosław Piliszek,

I remembered I set it to 1.26 to match the version of docker used by kolla. At that time, kolla is using docker 1.26. If it is safe to remove it or upgrade it right now, let's do it.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Hongbin Lu (<email address hidden>) on branch: master
Review: https://review.opendev.org/702065
Reason: duplicated with https://review.opendev.org/#/c/702034/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/702034
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=5e8431e0cc6d1e9f24f482b08e8429db77d21c11
Submitter: Zuul
Branch: master

commit 5e8431e0cc6d1e9f24f482b08e8429db77d21c11
Author: Radosław Piliszek <email address hidden>
Date: Fri Jan 10 21:06:15 2020 +0100

    Fix Zun Docker runtime selection

    Per [1] Docker uses API version 1.26 since daemon version 1.13.1.
    Kolla Ansible forced a limit on the Docker API version reported
    to Zun of 1.24 which was surprising users and preventing
    usage of Docker runtime selection via Zun.
    The default of Zun is 1.26 [2] which enables runtime selection.
    Zun does not use this parameter for anything else atm.

    [1] https://docs.docker.com/engine/api/v1.26/#section/Versioning
    [2] https://review.opendev.org/490794

    Change-Id: Ide6eb4c2b079cf35e633ad6a04db1587e40e7633
    Closes-bug: #1859176

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/702225

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/702226

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/702225
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=c6ff77a9c1187c08f88318b94d78ce1b9a1782f6
Submitter: Zuul
Branch: stable/train

commit c6ff77a9c1187c08f88318b94d78ce1b9a1782f6
Author: Radosław Piliszek <email address hidden>
Date: Fri Jan 10 21:06:15 2020 +0100

    Fix Zun Docker runtime selection

    Per [1] Docker uses API version 1.26 since daemon version 1.13.1.
    Kolla Ansible forced a limit on the Docker API version reported
    to Zun of 1.24 which was surprising users and preventing
    usage of Docker runtime selection via Zun.
    The default of Zun is 1.26 [2] which enables runtime selection.
    Zun does not use this parameter for anything else atm.

    [1] https://docs.docker.com/engine/api/v1.26/#section/Versioning
    [2] https://review.opendev.org/490794

    Change-Id: Ide6eb4c2b079cf35e633ad6a04db1587e40e7633
    Closes-bug: #1859176
    (cherry picked from commit 5e8431e0cc6d1e9f24f482b08e8429db77d21c11)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/702226
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=2a1e6e1c608f07be18d5aa4406dfb7744892e32e
Submitter: Zuul
Branch: stable/stein

commit 2a1e6e1c608f07be18d5aa4406dfb7744892e32e
Author: Radosław Piliszek <email address hidden>
Date: Fri Jan 10 21:06:15 2020 +0100

    Fix Zun Docker runtime selection

    Per [1] Docker uses API version 1.26 since daemon version 1.13.1.
    Kolla Ansible forced a limit on the Docker API version reported
    to Zun of 1.24 which was surprising users and preventing
    usage of Docker runtime selection via Zun.
    The default of Zun is 1.26 [2] which enables runtime selection.
    Zun does not use this parameter for anything else atm.

    [1] https://docs.docker.com/engine/api/v1.26/#section/Versioning
    [2] https://review.opendev.org/490794

    Change-Id: Ide6eb4c2b079cf35e633ad6a04db1587e40e7633
    Closes-bug: #1859176
    (cherry picked from commit 5e8431e0cc6d1e9f24f482b08e8429db77d21c11)
    (cherry picked from commit c6ff77a9c1187c08f88318b94d78ce1b9a1782f6)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.1 release.