(stable/stein) standalone-upgrade job fails in post-upgrade tasks for "rmtree failed: [Errno 16] Device or resource busy: '/var/lib/docker/containers"

[Marios Andreou]

The stable/stein tripleo-ci-centos-7-standalone-upgrade fails during the post-upgrade tasks (i.e. after completing upgrade-tasks & deploy-tasks) cleaning up docker - failing task is here [1] and examples are in [2]/[3] many more via [4]

Trace looks like:

 2019-04-25 02:19:36 | 2019-04-25 02:19:36.996 139981 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] fatal: [standalone]: FAILED! => {"changed": false, "msg": "rmtree failed: [Errno 16] Device or resource busy: '/var/lib/docker/containers/95417eccf15521349c34209057eccb49d806899b84d8d03bed512cabdbe4b445/shm'"}
 2019-04-25 02:19:36 | 2019-04-25 02:19:36.997 139981 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] 
 2019-04-25 02:19:36 | 2019-04-25 02:19:36.998 139981 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] NO MORE HOSTS LEFT *************************************************************
 2019-04-25 02:19:36 | 2019-04-25 02:19:36.999 139981 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] 
 2019-04-25 02:19:36 | 2019-04-25 02:19:37.000 139981 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] PLAY RECAP *********************************************************************
 2019-04-25 02:19:37 | 2019-04-25 02:19:37.000 139981 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] standalone : ok=92 changed=26 unreachable=0 failed=1
 2019-04-25 02:19:37 | 2019-04-25 02:19:37.000 139981 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] 

[1] https://opendev.org/openstack/tripleo-heat-templates/src/commit/9ce737e00491e93a2ce379f5ec74ccc217f8abac/deployment/podman/podman-baremetal-ansible.yaml#L120
[2] http://logs.openstack.org/44/654444/1/check/tripleo-ci-centos-7-standalone-upgrade/bb409b5/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz
[3] http://logs.openstack.org/35/655135/1/check/tripleo-ci-centos-7-standalone-upgrade/d4bff81/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz
[4] http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone-upgrade&branch=stable%2Fstein

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/655644

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/655646

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/656966

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/656967

[OpenStack Infra (hudson-openstack)]

Change abandoned by Marios Andreou (<email address hidden>) on branch: master
Review: https://review.opendev.org/655644
Reason: better suggestion at https://review.opendev.org/#/c/656966/1/deployment/podman/podman-baremetal-ansible.yaml

[OpenStack Infra (hudson-openstack)]

Change abandoned by Marios Andreou (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/655646
Reason: better suggestion at https://review.opendev.org/#/c/656966/1/deployment/podman/podman-baremetal-ansible.yaml

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

So on a reproducer environment the problem is that some directory are not unmounted properly where docker stop and are left on the host system. Those mount point prevent rm to work.

[root@standalone ~]# rm -rf /var/lib/docker 2>&1 │·······································
rm: cannot remove ‘/var/lib/docker/containers/1d10b03eb2ff51689784230e6f85de5202382a3b0b760a2747ee783c152cfd3d/shm’: Device or resource busy │·······································
rm: cannot remove ‘/var/lib/docker/containers/60054c23a52ce463edcbce9ba7266314aeda8facdcb4860fc9532b7f1128d759/shm’: Device or resource busy │·······································
rm: cannot remove ‘/var/lib/docker/containers/e290c3ea5d4d3d69d9a73e5274eb6659020f851fe96c9e16c10965bd740fd067/shm’: Device or resource busy │·······································
rm: cannot remove ‘/var/lib/docker/containers/2544b2fc7975632eab203bf78a152f32d4a0c5f6be7d9050e2f62a188245d808/shm’: Device or resource busy │·······································
rm: cannot remove ‘/var/lib/docker/overlay2/abe04cfd6459cab5da9fd6d8628c9a9304eb02a47d544fb53f60799a45fac1bc/merged’: Device or resource busy │·······································
rm: cannot remove ‘/var/lib/docker/overlay2/17bbf58868bb9795e392bb599a3ac8acf2ea04da104b6f3537807497532fa638/merged’: Device or resource busy │·······································
rm: cannot remove ‘/var/lib/docker/overlay2/85a3f3413ca05bfeb9bcc0b4482fdfb7afcc809a4792dfe72fe8abfe63dd7960/merged’: Device or resource busy │·······································
rm: cannot remove ‘/var/lib/docker/overlay2/90fc63bc4ff6b896c89951272a3ab2f360714a6a83e2a0f755370f1420b6784e/merged’: Device or resource busy │·······································
rm: cannot remove ‘/var/lib/docker/overlay2/53fa89987d4eca439114d730d2e30492ac187f2154aff3069c65b1621880868c/merged’: Device or resource busy

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/656966
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a8ec69941657975fe6a2b8616c3c729013a30dff
Submitter: Zuul
Branch: master

commit a8ec69941657975fe6a2b8616c3c729013a30dff
Author: Jose Luis Franco Arza <email address hidden>
Date: Fri May 3 17:33:54 2019 +0200

    Clean up leftover mount point after docker stop.

    Some mountpoints are left on the host after docker shut down. This
    seems to be an issue with docker, but couldn't trace it back to a
    specific docker bug.

    In the meantime to unblock the upgrade CI we make sure that every
    mount point under /var/lib/docker are umounted before deleting that
    directory.

    Note that we need to keep the order right, so that we do a depth first
    list so that umount can do leaves before root.

    Closes-Bug: #1826375
    Co-Authored-By: Sofer Athlan-Guyot <email address hidden>

    Change-Id: I03a065556caca4385bb8b28be0dfbe21addbf003

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/656967
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0152ce48ba2c5a9389180d2528dde50445ef5996
Submitter: Zuul
Branch: stable/stein

commit 0152ce48ba2c5a9389180d2528dde50445ef5996
Author: Jose Luis Franco Arza <email address hidden>
Date: Fri May 3 17:33:54 2019 +0200

    Clean up leftover mount point after docker stop.

    Some mountpoints are left on the host after docker shut down. This
    seems to be an issue with docker, but couldn't trace it back to a
    specific docker bug.

    In the meantime to unblock the upgrade CI we make sure that every
    mount point under /var/lib/docker are umounted before deleting that
    directory.

    Note that we need to keep the order right, so that we do a depth first
    list so that umount can do leaves before root.

    Closes-Bug: #1826375
    Co-Authored-By: Sofer Athlan-Guyot <email address hidden>
    (cherry picked from commit a8ec69941657975fe6a2b8616c3c729013a30dff)

    Change-Id: I03a065556caca4385bb8b28be0dfbe21addbf003

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.0 release.