Error device is busy

[Kevin]

Using Ubuntu 14.04 LTS fresh install.

git clone sahara-image-elements project. run

sudo -e venv -- sahara-image-create

3 of the images get created but then I get this error.

this was ran on the master branch.

I would expect all of the images that should be created would be.

dib-run-parts Thu Jun 23 22:16:25 CDT 2016
dib-run-parts Thu Jun 23 22:16:25 CDT 2016 00-clean-java-vars 0.001
dib-run-parts Thu Jun 23 22:16:25 CDT 2016 01-ccache 0.005
dib-run-parts Thu Jun 23 22:16:25 CDT 2016 51-bootloader 0.184
dib-run-parts Thu Jun 23 22:16:25 CDT 2016 99-remove-yum-repo-conf 0.015
dib-run-parts Thu Jun 23 22:16:25 CDT 2016 99-tidy-logs 0.059
dib-run-parts Thu Jun 23 22:16:25 CDT 2016
dib-run-parts Thu Jun 23 22:16:25 CDT 2016 --------------------- END PROFILING ---------------------
Unmount /tmp/dib_build.31jC32KD/mnt/sys
Unmount /tmp/dib_build.31jC32KD/mnt/proc
Unmount /tmp/dib_build.31jC32KD/mnt/dev/pts
Unmount /tmp/dib_build.31jC32KD/mnt/dev
Unmount /tmp/dib_build.31jC32KD/mnt
/dev/loop0: [0023]:3 (/tmp/dib_image.SDQPZmtQ/image.raw)
umount2: Device or resource busy
umount: /tmp/dib_build.31jC32KD: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount2: Device or resource busy
rm: cannot remove '/tmp/dib_build.31jC32KD': Device or resource busy
*** /tmp/dib_build.31jC32KD/mnt is not a directory
ERROR: InvocationError: '/home/j5321/git/sahara-image-elements/.tox/venv/bin/sahara-image-create'
___________________________ summary ___________________________
ERROR: venv: commands failed

[Vitalii Gridnev (vgridnev)]

Seems like your have issues with environment where images were built (luck of ram resources). Could you please try to set the following line before building images:

https://github.com/openstack/sahara-image-elements/blob/master/tools/gate/build-images#L4

export DIB_MIN_TMPFS=10

it's fixed our jobs some time ago.

[Vitalii Gridnev (vgridnev)]

Also, please don't use sudo -e venv -- sahara-image-create to build all images (it's better to build images you really need explicitly). We don't use sudo for building images, so sudo should be replaced with just tox, I guess

[Kevin (kvasko)]

Sorry, I just typed the command incorrectly.

sudo tox -e venv -- sahara-image-create

export DIB_MIN_TMPFS=10 does not correct the issue (I already tried it).

I don't think it should matter if I create 1 or all of the images, they should all work. The documentation implies that it should work by just running that command. But no matter,

running:

 export DIB_MIN_TMPFS=10
 sudo tox -e venv -- sahara-image-create -i centos -p ambari

Results in this error.

Unmount /tmp/dib_build.WlM4gCou/mnt/sys
Unmount /tmp/dib_build.WlM4gCou/mnt/proc
Unmount /tmp/dib_build.WlM4gCou/mnt/dev/pts
Unmount /tmp/dib_build.WlM4gCou/mnt/dev
Unmount /tmp/dib_build.WlM4gCou/mnt
/dev/loop0: [0021]:3 (/tmp/dib_image.7SwtJLjR/image.raw)
umount2: Device or resource busy
umount: /tmp/dib_build.WlM4gCou: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount2: Device or resource busy
rm: cannot remove '/tmp/dib_build.WlM4gCou': Device or resource busy
*** /tmp/dib_build.WlM4gCou/mnt is not a directory
ERROR: InvocationError: '/home/j5321/git/sahara-image-elements/.tox/venv/bin/sahara-image-create -i centos -p ambari'
___________________________________ summary ____________________________________
ERROR: venv: commands failed

[Vitalii Gridnev (vgridnev)]

Sudo is unneeded for running images. Now it can produce with not available caches during image creation, because cache was created with sudo permissions.

You may try larger value of DIB_MIN_TMPFS=10, for example 20.

so, I expect that everything should be working with
X=(20,30..)
export DIB_MIN_TMPFS=X
tox -e venv -- sahara-image-create -i centos -p ambari

[Kevin (kvasko)]

So it looks like there is an issue with the "centos" creation. Ubuntu, Fedora and centos7 images get created successfully. This tool seems to be extremely flaky.

sudo tox -e venv -- sahara-image-create -i centos7 -p ambari

or

tox -e venv -- sahara-image-create -i centos7 -p ambari

work

The following does not work and I get the same error message as originally poste:

tox -e venv --sahara-image-create -i centos -p ambari //FAILS with above error (with or without export DIB_MIN_TIMPFS=20)

See below for working CentOS7 working...
----------------------

dib-run-parts Fri Jun 24 09:32:23 CDT 2016 --------------------- END PROFILING ---------------------
Unmount /tmp/dib_build.IJ1WhnXO/mnt/sys
Unmount /tmp/dib_build.IJ1WhnXO/mnt/proc
Unmount /tmp/dib_build.IJ1WhnXO/mnt/dev/pts
Unmount /tmp/dib_build.IJ1WhnXO/mnt/dev
Unmount /tmp/dib_build.IJ1WhnXO/mnt
/dev/loop0: [0023]:3 (/tmp/dib_image.4BTF2TTf/image.raw)
Converting image using qemu-img convert
Old image found. Renaming it to centos7-sahara-ambari-2016.06.24-09.34.21.qcow2
Image file centos7-sahara-ambari.qcow2 created...
___________________________________ summary ____________________________________
  venv: commands succeeded
  congratulations :)

[Vitalii Gridnev (vgridnev)]

Kevin, I agreed that sahara-image-create is flaky instrument, that is why we are preparing images right after release and publishing them. Also, right now we trying to create new solution for building and preparing images.

Could you please describe environment which was used to reproduce bug? I'm usualy creating m1.large VM to create images.

BTW, we also have gate jobs enabled for upstream of sahara-image-elements, and seems like all of them are passed (them are also running on ubuntu vms):

https://review.openstack.org/#/c/328625/

[Kevin (kvasko)]

Downloaded 14.04LTS ISO from Ubuntu website

Created a Virtual Machine with VirtualBox 4.x on my Windows host PC.

sudo apt-get install git
sudo apt-get install python-pip
sudo pip install tox
sudo apt-get install qemu kpartx
tox -e venv -- sahara-image-create -i centos -p ambari

centos7 seems to work though.

seems to result in the error in original post.

[Vitalii Gridnev (vgridnev)]

what values of cpu, ram, and disk this VM has?

[Kevin (kvasko)]

The VM I created in VirtualBox has 4 CPUs, 8 GB ram and 80GB of disk space.

[Vitalii Gridnev (vgridnev)]

Just tried on my vm with same configuration and DIB_MIN_TMPFS=10 and bug is not reproduced: image was created successfully.

Could you please attach the full log since start of executing script (not just finish of logs)?

[Kevin (kvasko)]

I think I see what happened...

sudo apt-get install git
sudo apt-get install python-pip
sudo pip install tox
sudo apt-get install qemu kpartx
export DIB_MIN_TMPFS=10
sudo tox -e venv -- sahara-image-create -i centos -p ambari // DOES NOT WORK
tox -e venv -- sahara-image-create -i centos -p ambari // WORKS

still don't quite understand how a higher access account could generate the issue but a lower leveled account works.

[Vitalii Gridnev (vgridnev)]

Actually, I think that when you running with "sudo tox ..." it doesn't mean that every script from imagebuilder will executed with sudo permissions. So, because of that we just recommending to run image building just as "tox ..".

Ok, seems like it can be closed now, thanks for reporting the bug.