Image Builder: modify_image encounters OSError for some packages

[Para Siva]

I've seen similar behaviour for a couple of packages whoopsie first and then tgt and the trace starts from modify_image as follows.

Traceback (most recent call last):
  File "./image-builder/imagebuilder/run_worker.py", line 59, in on_message
    image_id = modify_image(image, repos, series, packages, status_cb)
  File "/srv/imagebuild_worker/image-builder/imagebuilder/cloud_image.py", line 204, in modify_image
    return glance_filename
  File "/srv/imagebuild_worker/ci-utils/ci_utils/__init__.py", line 30, in __exit__
    shutil.rmtree(self.path)
  File "/usr/lib/python2.7/shutil.py", line 245, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 245, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 245, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 254, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python2.7/shutil.py", line 252, in rmtree
    os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/tmp/tmpYL9Vsn/mountpoint/dev/pts'

http://pastebin.ubuntu.com/7039416/ for whoopsie,
http://pastebin.ubuntu.com/7049101/ for tgt (https://swift.canonistack.canonical.com/v1/AUTH_b4f126adde1e45bd8d10e3ee3b13490b/ticket-1/imagebuild-output.log)

So I created a trusty chroot env on my trusty laptop and tried to install these two packages and they failed in post install 'setting up' stage.

It could be that I am using absurd packages but because the emphasis being on kernel packages not sure if we see more issues like this in the chroot env.

[Para Siva (psivaa)]

btw, it was my pkg selections that's absurd, i dint mean anything bad about the packages themselves :)

What's more concerning is that once this happens with one of these packages, the imagebuild_worker has to be rebooted for image building to succeed with any other packages.

[Para Siva (psivaa)]

@Paul: I think what's going on here. I'll take this bug for now.

[Para Siva (psivaa)]

This is happening with packages that start their background services and processes automatically on installation. The unmounting/ removing the chroot dirs get permission denied because there are used by those background processes. lsof of the concerned dir's lists the daemon processes of the packages being installed.

Killing those processes alone doesn't make the dir's removable. One needs to logout and login back to make the dir's completely removable.

So over the weekend, I tried to workaround this by preventing the processes from running during installation itslef by pointing to invoke-rc.d, initctl, etc as given in http://askubuntu.com/questions/74061/install-packages-without-starting-background-processes-and-services. I have already tried running "chroot /tmp/tmpPrgN44/mountpoint /usr/bin/env - PATH=/root/fake:$PATH /usr/bin/apt-get install" and subprocess.Popen([cmd] , env={"PATH": "/root/fake:$PATH"}) but both fail with 127 in the chrooted env.

@Paul: I think I have exhausted all my avenues here to fix the issue. Hoping you could find a way to workaround this I am assigning back to you.

[Evan (ev)]

You're on the right track. Ubiquity has code for diverting invoke-rc.d, upstart, start-stop-daemon, and others. I'd suggest using that as a source.

It would be helpful to see the full logs of this incorporated with the ubiquity code and failing, if it fails.