vm-builder needs trusty suite

[Chris J Arges]

[Impact]
When using vm-builder it should be able to specify 'trusty' as a suite.

[Test Case]
sudo vmbuilder kvm ubuntu --suite trusty

[Regression Potential]
This just adds another release name.

[Serge Hallyn (serge-hallyn)]

Thanks for submitting this bug.

[Brian Murray (brian-murray)]

Hello Chris, or anyone else affected,

Accepted vm-builder into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/vm-builder/0.12.4+bzr488-0ubuntu1.13.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Chris, or anyone else affected,

Accepted vm-builder into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/vm-builder/0.12.4+bzr477-0ubuntu5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Beltran (beltran)]

In addition to the files, it seems it is also necessary to mount /proc before installing the kernel for Saucy or Trusty.

/usr/share/pyshared/VMBuilder/plugins/ubuntu/dapper.py
@@ -300,7 +300,9 @@
         return (mirror, updates_mirror, security_mirror)

     def install_kernel(self, destdir):
+ self.run_in_target('mount', '-t', 'proc', 'proc', '/proc')
         run_cmd('chroot', destdir, 'apt-get', '--force-yes', '-y', 'install', self.kernel_name(), env={ 'DEBIAN_FRONTEND' : 'noninteractive' })
+ run_cmd('umount', '%s/proc' % self.context.chroot_dir)

     def install_grub(self, chroot_dir):
         self.install_from_template('/etc/kernel-img.conf', 'kernelimg', { 'updategrub' : self.updategrub })

[Serge Hallyn (serge-hallyn)]

Hi Beltran,

Which release are you doing this on? Trusty vm builds fine for me on
trusty.

[AmirAli Akbari (amiraliakbari)]

Thanks Brian, updating to proposed sources (vmbuilder version 0.12.4.r475) solved the problem for me on precise.

Meanwhile, in this revision, when calling vmbuilder --version, a python error is raised. The cause is that 'revno' returned by the VMBuilder.get_version_info is of type str, not int.

VMBuilder/contrib/cli.py:282
- print ('%(major)d.%(minor)d.%(micro)s.r%(revno)d' %
+ print ('%(major)d.%(minor)d.%(micro)s.r%(revno)s' %

[Serge Hallyn (serge-hallyn)]

Hi Amir,

please open a new bug for the crash you are seeing. Indeed
revno should be an int not str in vcsersion.py

[Johan Smits (johan-smits)]

Tested 0.12.4+bzr477-0ubuntu5 on precise (64 bits) did not work for me.
It stalls on:
....
2014-04-30 09:22:56,349 DEBUG : I: Configuring ubuntu-minimal...
2014-04-30 09:22:56,350 DEBUG : I: Configuring libc-bin...
2014-04-30 09:22:56,368 DEBUG : I: Configuring initramfs-tools...
2014-04-30 09:22:56,387 DEBUG : I: Base system installed successfully.
^CTraceback (most recent call last):
  File "/usr/bin/ubuntu-vm-builder", line 24, in <module>
    uvb.main()
  File "/usr/lib/python2.7/dist-packages/VMBuilder/contrib/cli.py", line 216, in main
    distro.build_chroot()
  File "/usr/lib/python2.7/dist-packages/VMBuilder/distro.py", line 83, in build_chroot
    self.call_hooks('bootstrap')
  File "/usr/lib/python2.7/dist-packages/VMBuilder/distro.py", line 67, in call_hooks
    call_hooks(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/VMBuilder/util.py", line 165, in call_hooks
    getattr(context, func, log_no_such_method)(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/VMBuilder/plugins/ubuntu/distro.py", line 136, in bootstrap
    self.suite.debootstrap()
  File "/usr/lib/python2.7/dist-packages/VMBuilder/plugins/ubuntu/dapper.py", line 269, in debootstrap
    run_cmd(*cmd, **kwargs)
  File "/usr/lib/python2.7/dist-packages/VMBuilder/util.py", line 113, in run_cmd
    fds = select.select([x.file for x in [mystdout, mystderr] if not x.closed], [], [])[0]
KeyboardInterrupt

ps ax shows:
5522 pts/2 S+ 0:00 /usr/bin/python /usr/bin/ubuntu-vm-builder kvm trusty --
 5524 pts/2 Z+ 0:00 [debootstrap] <defunct>

Building precise builds works ok.

[EJ Bensing (ej-x)]

I'm seeing the same error as Johan. Running 0.12.4+bzr477-0ubuntu5 on precise (64 bits)

[Brian Murray (brian-murray)]

Did you get the same error trying to build a saucy system?

[EJ Bensing (ej-x)]

I am seeing the same problem when I try with saucy.

[Brian Murray (brian-murray)]

Well then, that seems independent of the SRU then. Could you check and see if there is an existing bug report about vm-builder failing to build Saucy and if there isn't report one and mention it here? Thanks!

[EJ Bensing (ej-x)]

I did some searching around the bug tracker + google and couldn't find anything. This is my first time poking around the bug tracker here though, so it is possible I missed a report.

[Tobias Wolter (towo)]

Same as the others are seeing for trusty.

[Tobias Wolter (towo)]

Correction, not the same, but:

^CTraceback (most recent call last):
  File "/usr/bin/vmbuilder", line 24, in <module>
    cli.main()
  File "/usr/lib/python2.7/dist-packages/VMBuilder/contrib/cli.py", line 216, in main
    distro.build_chroot()
  File "/usr/lib/python2.7/dist-packages/VMBuilder/distro.py", line 83, in build_chroot
    self.call_hooks('bootstrap')
  File "/usr/lib/python2.7/dist-packages/VMBuilder/distro.py", line 67, in call_hooks
    call_hooks(self, *args, **kwargs)
KeyboardInterrupt

I'll have a look later if I can find something to patch there

[Brian Candler (b-candler)]

I see hang at "Base system installed successfully" too, building Trusty VM under Precise 64-bit.

Last three lines of ps auxwww are:

root 20430 0.1 0.0 40060 9716 pts/2 S+ 10:36 0:00 /usr/bin/python /usr/bin/vmbuilder kvm ubuntu --suite trusty --hostname trusty --mem 512 --debug --dest /var/lib/libvirt/images/trusty
root 20432 0.0 0.0 0 0 pts/2 Z+ 10:36 0:00 [debootstrap] <defunct>
root 31685 0.0 0.0 9892 1180 ? Ss 10:37 0:00 udevd --daemon

There is a zombie debootstrap.

# pstree -s 20432
init───sshd───sshd───sshd───bash───sudo───bash───vmbuilder───debootstrap

# ps -o pid,ppid -C debootstrap
  PID PPID
20432 20430

Same backtrace from ctrl-C:

^CTraceback (most recent call last):
  File "/usr/bin/vmbuilder", line 24, in <module>
    cli.main()
  File "/usr/lib/python2.7/dist-packages/VMBuilder/contrib/cli.py", line 216, in main
    distro.build_chroot()
  File "/usr/lib/python2.7/dist-packages/VMBuilder/distro.py", line 83, in build_chroot
    self.call_hooks('bootstrap')
  File "/usr/lib/python2.7/dist-packages/VMBuilder/distro.py", line 67, in call_hooks
    call_hooks(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/VMBuilder/util.py", line 165, in call_hooks
    getattr(context, func, log_no_such_method)(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/VMBuilder/plugins/ubuntu/distro.py", line 136, in bootstrap
    self.suite.debootstrap()
  File "/usr/lib/python2.7/dist-packages/VMBuilder/plugins/ubuntu/dapper.py", line 269, in debootstrap
    run_cmd(*cmd, **kwargs)
  File "/usr/lib/python2.7/dist-packages/VMBuilder/util.py", line 113, in run_cmd
    fds = select.select([x.file for x in [mystdout, mystderr] if not x.closed], [], [])[0]
KeyboardInterrupt

So it's definitely a question of python waiting for data from the child, but the child has gone away.

[Brian Candler (b-candler)]

There is a nasty race condition in the code, which is easily fixed:

--- /usr/share/pyshared/VMBuilder/util.py.orig 2010-06-10 17:20:58.000000000 +0000
+++ /usr/share/pyshared/VMBuilder/util.py 2014-06-19 10:57:09.723017475 +0000
@@ -108,9 +108,12 @@
     mystdout = NonBlockingFile(proc.stdout, logfunc=logging.debug)
     mystderr = NonBlockingFile(proc.stderr, logfunc=(ignore_fail and logging.debug or logging.info))

- while not (mystdout.closed and mystderr.closed):
+ while True:
+ fdset = [x.file for x in [mystdout, mystderr] if not x.closed]
+ if not fdset:
+ break
         # Block until either of them has something to offer
- fds = select.select([x.file for x in [mystdout, mystderr] if not x.closed], [], [])[0]
+ fds = select.select(fdset, [], [])[0]
         for fp in [mystderr, mystdout]:
             if fp.file in fds:
                 fp.process_input()

Unfortunately that doesn't fix the problem :-(

Adding debugging shows that mystdout remains open, but does not show ready to read from select().

Oddly, it seems that mystderr.closed becomes true almost immediately after starting debootstrap.

2014-06-19 11:16:11,884 DEBUG : Calling bootstrap method in context plugin VMBuilder.plugins.ubuntu.distro.
2014-06-19 11:16:11,884 DEBUG : ['/usr/sbin/debootstrap', '--arch=i386', 'trusty', '/tmp/tmpBQAxhc', 'http://apt.ws.nsrc.org:3142/archive.ubuntu.com/ubuntu']
*** Starting ['/usr/sbin/debootstrap', '--arch=i386', 'trusty', '/tmp/tmpBQAxhc', 'http://apt.ws.nsrc.org:3142/archive.ubuntu.com/ubuntu'] ***
mystdout=<VMBuilder.util.NonBlockingFile object at 0x7fcade825550>, closed=False
mystderr=<VMBuilder.util.NonBlockingFile object at 0x7fcade825610>, closed=False
fdset=[<open file '<fdopen>', mode 'rb' at 0x7fcadea726f0>, <open file '<fdopen>', mode 'rb' at 0x7fcadea72780>]
selecting...
fds=[<open file '<fdopen>', mode 'rb' at 0x7fcadea72780>]
mystdout=<VMBuilder.util.NonBlockingFile object at 0x7fcade825550>, closed=False
mystderr=<VMBuilder.util.NonBlockingFile object at 0x7fcade825610>, closed=True
fdset=[<open file '<fdopen>', mode 'rb' at 0x7fcadea726f0>]
selecting...
fds=[<open file '<fdopen>', mode 'rb' at 0x7fcadea726f0>]

It could be true that debootstrap closes stderr immediately of course.

[Brian Candler (b-candler)]

I can replicate this issue with a standalone python script.
http://pastebin.com/vTREYc9n

It hangs at:
read stdout: 'I: Base system installed successfully.\n'

Furthermore, if I add a timeout to select and force it to read anyway, I get an IOError:

read stdout: [Errno 11] Resource temporarily unavailable

It's easy to eliminate the zombie child. You can reap the child explicitly by adding this to the main loop:

    sys.stderr.write("poll=%s\n" % repr(proc.poll()))

However even after the child has terminated, select() still says there's no data to read, and read() gives EAGAIN.

Note: changing the code to use proc.communicate() also hangs and leaves a zombie child. And that uses threads rather than non-blocking sockets.

This looks very much like a bug in python to me.

[Brian Candler (b-candler)]

FYI: the system I'm using has kernel 3.8.0-42-generic, i.e. it was installed at the time when the "raring enablement stack" was active.

I can now reproduce with just a few lines:

---
import subprocess

args = ['/usr/sbin/debootstrap', '--arch=i386', 'trusty', '/tmp/tmpBQAxhc', 'http://archive.ubuntu.com/ubuntu']
stdin_arg = file('/dev/null', 'r')
proc = subprocess.Popen(args, stdin=stdin_arg, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

print repr(proc.communicate())
---

strace shows the hang here:

...
read(4, "I: Configuring initramfs-tools.."..., 1224) = 34
read(4, "I: Base system installed success"..., 1190) = 39
read(4, 0xd61919, 1151) = ? ERESTARTSYS (To be restarted)
--- SIGCHLD (Child exited) @ 0 (0) ---
read(4,

[Brian Candler (b-candler)]

Argh. Scrub what I said about python. I can reproduce without python!

# /usr/sbin/debootstrap --arch=i386 trusty /tmp/tmpBQAxhc http://archive.ubuntu.com/ubuntu | strace cat >db.out
...
read(0, "I: Configuring initramfs-tools.."..., 32768) = 34
write(1, "I: Configuring initramfs-tools.."..., 34) = 34
read(0, "I: Base system installed success"..., 32768) = 39
write(1, "I: Base system installed success"..., 39) = 39
read(0,

It hangs here - and there's no zombie child, so can't be debootstrap's fault. This is something really weird with pipes.

[Brian Candler (b-candler)]

OK, I found it.

During debootstrap run, a process "udevd --daemon" is started. It persists when debootstrap gets to the end. If you kill it, the process reading from debootstrap's stdout is able to terminate.

My guess is that stdout is being passed to that daemon, and it's holding it open.

No idea of the fix though. Why is udevd being started in the first place? If it's being started as a daemon, why is it not closing its stdout? And if there is a reason for running it, shouldn't it be stopped before debootstrap finishes?

[Serge Hallyn (serge-hallyn)]

Please open a new bug against debootstraph for this, clearly specifying
your release and any ppa's (like cloud archive) you have enabled. I
believe this was fixed by the 1.0.52 version, which we can SRU back to
precise if needed.

[Brian Candler (b-candler)]

I have raised these:

Bug #1332155: debootstrap launches spurious udevd --daemon
Bug #1332165: udevd --daemon does not close stdout

[Brian Candler (b-candler)]

The race condition in comment #17 is only true if mystdout.closed can become True asynchronously before we have called mystdout.close(). I'm not sure if there's any possibility of that.

[Jonathan Jogenfors (etnoy)]

For me the problem was solved by updating python-vmbuilder to the latest version.

[Mathew Hodson (mhodson)]

Jonathon, which version of the package did you use?

[Bartosz Kosiorek (gang65)]

It is working perfectly for me (Precise)

[Launchpad Janitor (janitor)]

This bug was fixed in the package vm-builder - 0.12.4+bzr477-0ubuntu5

---------------
vm-builder (0.12.4+bzr477-0ubuntu5) precise-proposed; urgency=medium

  * Add trusty to the list of valid ubuntu suites. (LP: #1287943)
 -- Serge Hallyn <email address hidden> Tue, 04 Mar 2014 16:52:51 -0600

[Chris J Arges (arges)]

The verification of the Stable Release Update for vm-builder has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.