Failed to unmount on clean up when building Centos8 IPA image on Docker

[Gudrutis]

At CERN, we use privileged docker containers to setup and isolate diskimage-builder environments. Lately, we noticed that when building Centos8 based IPA image, the build process fails to clean up with the following error:

```
2020-07-10 10:16:30.967 | umount: /tmp/dib_image.qqWDHy7D: target is busy.
2020-07-10 10:16:30.967 | (In some cases useful info about processes that use
2020-07-10 10:16:30.967 | the device is found by lsof(8) or fuser(1))
```

While the image is built successfully, this is still inconvenient.
I tried various different environments, but I still get this error:
- docker image based on any OS centos7 and centos8
- using docker host based on centos7 and centos8 or ubuntu
- used both diskimage-builder.git@2.37.1 and diskimage-builder.git@3.1.0

This issues does not happen when building Centos7 image.

I am only able to build an Centos8 image without getting an error if I do not use docker and execute diskimage-builder on the server directly, but this does not give us the isolation we want.

This is how we run docker:

```

docker run -v /dev:/dev --privileged --mount source=images,target=/ironic-image-creation/outputimage -it gitlab-registry.cern.ch/cloud-infrastructure/ironic-python-agent-builder:cc8 /bin/bash

```

This is the script how we execute the command:
```

mkdir -p /ironic-image-creation/outputimage/$TEMP_FOLDER
export ELEMENTS_PATH="$(dirname ${SCRIPT_DIRECTORY})/dib"
# Setup enviroment
pip install -U pip || true
pip install virtualenv || true
virtualenv $SCRIPT_DIRECTORY/venv || true
source $SCRIPT_DIRECTORY/venv/bin/activate
pip install git+https://opendev.org/openstack/diskimage-builder.git@2.37.1
# Debug
export DIB_DEV_USER_AUTHORIZED_KEYS="${SCRIPT_DIRECTORY}/authorized_keys"
export DIB_DEV_USER_USERNAME=admin
export DIB_DEV_USER_PWDLESS_SUDO=yes
export DIB_DEBUG_TRACE=1
export BUILD_LOG=/ironic-image-creation/outputimage/$TEMP_FOLDER/build.log
# Configuration of DIB elements
export DIB_REPOREF_ironic_python_agent=$IPA_BRANCH
export DIB_REPOREF_requirements=$REQUIREMENTS_BRANCH
# To be able to build latest release, use python 3
export DIB_PYTHON_VERSION=3
export DIB_RELEASE=8
export DIB_DISTRIBUTION_MIRROR=http://linuxsoft.cern.ch/cern/centos/

disk-image-create -o /ironic-image-creation/outputimage/$TEMP_FOLDER/$TEMP_FOLDER \
    devuser \
    disable-selinux \
    centos \
    ironic-python-agent-ramdisk \
    ironic-hwmanager-cern \
    local-config \
    cleanup-kernel-initrd \
    ironic-rsyslog \
    mellanox \
    development-pkgs \
    docker-ce | tee $BUILD_LOG

```

Let us know if there is solution to this problem or another way to isolate and sandbox the build process.

[Radosław Piliszek (yoctozepto)]

Did you have a chance to check what is keeping tmp busy?

[Gudrutis (zyrus123)]

[root@677741bc63ef tmp]# lsof +D /tmp/
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 1 root cwd DIR 0,54 169 161512418 /tmp
lsof 1175 root cwd DIR 0,54 169 161512418 /tmp
lsof 1176 root cwd DIR 0,54 169 161512418 /tmp
[root@677741bc63ef tmp]# rm -rf /tmp/dib_image.4Tm6LZVK/
rm: cannot remove ‘/tmp/dib_image.4Tm6LZVK/’: Device or resource busy

[Gudrutis (zyrus123)]

I think the script sometimes forgets to close the loopback file, which prevents the umount.

```
[root@4ec4a99a5d5d tmp]# losetup -a
/dev/loop1: [0062]:668710 (/tmp/dib_image.Uau63lur/image0.raw)
/dev/loop0: [0054]:242773 (/image0.raw)
[root@4ec4a99a5d5d tmp]# losetup -d /dev/loop1
[root@4ec4a99a5d5d tmp]# losetup -a
/dev/loop1: [0062]:668710 (/tmp/dib_image.Uau63lur/image0.raw)
/dev/loop0: [0054]:242773 (/image0.raw)
[root@4ec4a99a5d5d tmp]# umount /tmp/dib_image.Uau63lur/image0.raw
[root@4ec4a99a5d5d tmp]# losetup -a
/dev/loop0: [0054]:242773 (/image0.raw)
...
[root@4ec4a99a5d5d tmp]# umount /tmp/dib_image.Uau63lur/ # unmount successful
[root@4ec4a99a5d5d tmp]# rm -rf dib_image.*

```