docker build environment doesn't stop cleanly

[Scott Little]

Brief Description
-----------------
Docker build container doesn't stop/exit cleanly.
Docker images can't be cleaned up.
Consumption of system resource (disk).
Reboot and significant manual intervention to clean up.

Severity
--------
Major: Consumes system resources

Steps to Reproduce
------------------

docker build --build-arg MYUID=${MYUID} --build-arg MYUNAME=${MYUNAME} --ulimit core=0 --network host --tag $DOCKER_BUILD_IMAGE --file ${DOCKERFILE} .

docker run -it --rm --name ${DOCKER_BUILD_ID} --detach --volume $(readlink -f /localdisk):/localdisk --volume ${DOCKER_BUILD_WORKDIR}/centos-mirror-tools/output:/import/mirrors/CentOS:ro --volume /sys/fs/cgroup:/sys/fs/cgroup:ro --volume ~/.ssh:/mySSH:ro -e "container=docker" -e MYUNAME=${MYUNAME} --privileged=true --security-opt seccomp=unconfined ${DOCKER_BUILD_IMAGE}

docker stop ${DOCKER_BUILD_ID}

docker image rm $DOCKER_BUILD_IMAGE

Expected Behavior
------------------
Build container shuts down cleanly, and the image is auto removed.

'docker image rm' reports 'No such container'

Actual Behavior
----------------
Container is stuck in state 'Removal In Progress'

'docker image rm' reposts ...

unable to remove repository reference "..." (must force) - container e3c76f2e30ac is using its referenced image d8e4d73c1822

Reproducibility
---------------
Reproducible

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
2019-06-17

Last Pass
---------
N/A

Timestamp/Logs
--------------
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e3c76f2e30ac slittle1-master-stx-new-url-builder "/bin/sh -c /usr/sbi…" 6 hours ago Removal In Progress slittle1-master-stx-new-url-build

Test Activity
-------------
build

[Scott Little (slittle1)]

Our docker builder runs services under systemd.
Systemd requires SIGRTMIN+3 to trigger a clean exit.
Sigterm is ignored.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/666440

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/666440
Committed: https://git.openstack.org/cgit/starlingx/tools/commit/?id=de2ee28404b66efeb3deb53f227358579c165f2c
Submitter: Zuul
Branch: master

commit de2ee28404b66efeb3deb53f227358579c165f2c
Author: Scott Little <email address hidden>
Date: Wed Jun 19 16:24:57 2019 -0400

    Docker build environment does not exit cleanly.

    Our docker builder runs services under systemd.
    Systemd requires SIGRTMIN+3 to trigger a clean exit.
    The 'docker stop' command will send SIGTERM by default,
    which systemd ignores.

    Failure to exit cleanly results in mounts being left in
    place, and ultimately the container will get stuck in
    'Removal in progress' state, and a docker image that
    can't be removed.

    Change-Id: I28edb021a4f9c8941a7be57bfbb2ffa34b15ccd7
    Closes-Bug: 1833467
    Signed-off-by: Scott Little <email address hidden>

[Ghada Khalil (gkhalil)]

Build robustness; marking as stx.2.0 as the fix was committed for this release.