container not delete

[lxm]

containers can be deleted in zun successfully , but the containers on compute node are still running status

we got logs as follows

2019-07-28 23:58:40.141 45285 DEBUG zun.compute.manager [req-17f7705e-eee5-4133-bef0-d678eaa9d8e4 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Showing container: 0944adf7-e9a6-43ac-95d8-d685fd0bd2ec container_show /usr/lib/python2.7/site-packages/zun/compute/manager.py:568
2019-07-28 23:58:40.143 45285 DEBUG docker.utils.config [req-17f7705e-eee5-4133-bef0-d678eaa9d8e4 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Trying paths: ['/var/lib/zun/.docker/config.json', '/var/lib/zun/.dockercfg'] find_config_file /usr/lib/python2.7/site-packages/docker/utils/config.py:21
2019-07-28 23:58:40.143 45285 DEBUG docker.utils.config [req-17f7705e-eee5-4133-bef0-d678eaa9d8e4 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Found file at path: /var/lib/zun/.docker/config.json find_config_file /usr/lib/python2.7/site-packages/docker/utils/config.py:25
2019-07-28 23:58:40.144 45285 DEBUG docker.auth [req-17f7705e-eee5-4133-bef0-d678eaa9d8e4 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Found 'auths' section load_config /usr/lib/python2.7/site-packages/docker/auth.py:256
2019-07-28 23:58:40.145 45285 DEBUG docker.auth [req-17f7705e-eee5-4133-bef0-d678eaa9d8e4 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Found entry (registry=u'registry.honyasec.com', username=u'admin') parse_auth /usr/lib/python2.7/site-packages/docker/auth.py:218
2019-07-28 23:58:41.575 45285 DEBUG oslo_concurrency.lockutils [req-437351b4-4771-48c4-9278-9e8a23f6168b - - - - -] Lock "0944adf7-e9a6-43ac-95d8-d685fd0bd2ec" released by "zun.compute.manager.do_container_create" :: held 6.534s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339
2019-07-28 23:58:41.576 45285 DEBUG oslo_concurrency.lockutils [req-755ef353-2ac8-4209-899d-0fc6d6cadbb6 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Lock "0944adf7-e9a6-43ac-95d8-d685fd0bd2ec" acquired by "zun.compute.manager.do_container_delete" :: waited 5.646s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327
2019-07-28 23:58:41.576 45285 DEBUG zun.compute.manager [req-755ef353-2ac8-4209-899d-0fc6d6cadbb6 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Deleting container: 0944adf7-e9a6-43ac-95d8-d685fd0bd2ec _do_container_delete /usr/lib/python2.7/site-packages/zun/compute/manager.py:501
2019-07-28 23:58:41.577 45285 DEBUG zun.compute.manager [req-755ef353-2ac8-4209-899d-0fc6d6cadbb6 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Skip updating container task state to container_deleting because its current task state is: container_creating _update_task_state /usr/lib/python2.7/site-packages/zun/compute/manager.py:274
2019-07-28 23:58:41.578 45285 DEBUG docker.utils.config [req-755ef353-2ac8-4209-899d-0fc6d6cadbb6 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Trying paths: ['/var/lib/zun/.docker/config.json', '/var/lib/zun/.dockercfg'] find_config_file /usr/lib/python2.7/site-packages/docker/utils/config.py:21
2019-07-28 23:58:41.579 45285 DEBUG docker.utils.config [req-755ef353-2ac8-4209-899d-0fc6d6cadbb6 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Found file at path: /var/lib/zun/.docker/config.json find_config_file /usr/lib/python2.7/site-packages/docker/utils/config.py:25
2019-07-28 23:58:41.580 45285 DEBUG docker.auth [req-755ef353-2ac8-4209-899d-0fc6d6cadbb6 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Found 'auths' section load_config /usr/lib/python2.7/site-packages/docker/auth.py:256
2019-07-28 23:58:41.580 45285 DEBUG docker.auth [req-755ef353-2ac8-4209-899d-0fc6d6cadbb6 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Found entry (registry=u'registry.honyasec.com', username=u'admin') parse_auth /usr/lib/python2.7/site-packages/docker/auth.py:218
2019-07-28 23:58:41.594 45285 DEBUG oslo_concurrency.lockutils [req-755ef353-2ac8-4209-899d-0fc6d6cadbb6 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Lock "compute_resources" acquired by "zun.compute.compute_node_tracker.remove_usage_from_container" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327
2019-07-28 23:58:41.617 45285 DEBUG oslo_concurrency.lockutils [req-755ef353-2ac8-4209-899d-0fc6d6cadbb6 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Lock "compute_resources" released by "zun.compute.compute_node_tracker.remove_usage_from_container" :: held 0.023s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339
2019-07-28 23:58:41.618 45285 DEBUG oslo_concurrency.lockutils [req-755ef353-2ac8-4209-899d-0fc6d6cadbb6 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Lock "0944adf7-e9a6-43ac-95d8-d685fd0bd2ec" released by "zun.compute.manager.do_container_delete" :: held 0.043s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339

[hongbin (hongbin034)]

@lxm,

A couple of clarifying questions:

* How did you install Zun (i.e. devstack, kolla, manual install).
* Which version of Zun you are using (i.e. master, stable/stein, etc.)
* Is that a random error, or a steadily reproducible error? If latter, could you provide the reproducing steps?

[lxm (lxm-xupt)]

install method: manual install
version: lastest master on github (418f383d390443ecb5e5f7adb181d2c887951536)
it's a random error, but if we delete the container immediately after the container status became running ,the error occurs frequently

[lxm (lxm-xupt)]

in addition, i tried to watch the docker daemon logs, it seems that zun-compute never called the delete api of the docker damemon

[hongbin (hongbin034)]

Right,

This entry cautch my eye:

2019-07-28 23:58:40.143 45285 DEBUG docker.utils.config [req-17f7705e-eee5-4133-bef0-d678eaa9d8e4 3740ad4ed7ed40d68719c94f26fd0c5e 9d854a5d11884c84bfd5fdce35eeaae4 default - -] Found file at path: /var/lib/zun/.docker/config.json

Could you post this file or tell me what is inside: /var/lib/zun/.docker/config.json (please make sure to remove sensitive information like password, private key, etc.).

[lxm (lxm-xupt)]

{u'auths': {u'registry.xxxx.com': {u'auth': u'xxxxx'}}, u'HttpHeaders': {u'User-Agent': u'Docker-Client/18.06.1-ce (linux)'}}

[hongbin (hongbin034)]

@lxm,

I tried to reproduce but it looks I cannot. Could you provide more information about how to reproduce? For example, what is the command you used to create the container, and what is the command you use to delete the container. In addition, provide the full log of zun-compute and docker would help.