linaro-media-create hangs oneiric with the attached rootfs

[Mike Carifio]

On oneiric i386, I'm building an image file with the following command:

linaro-media-create --image-file build.img --hwpack-force-yes --dev mx53loco --binary rootfs.tar.gz --hwpack hwpack.tar.gz

[If the rootfs and hwpack are needed for debugging, I can provide them.]

I'm using linaro-image-tools '2011.12-0ubuntu1~linaro1'.

linaro-media-create consistently gets to the "Populating ... this will take a few minutes ..." message and then hangs my laptop. In /var/log/syslog, I consistently see this message:

init: /tmp/tmp<random>/binary/filesyste.dir/etc/init/plymouth-splash.conf: Unable to reload configuration after override deletion.
init: /tmp/tmp<random>/binary/filesyste.dir/etc/init: Configuration directory deleted
init: file.c:110: Unhandled error from nib_file_read:No such file or directory
init: Caught abort, core dumped
init: file.c:110: Unhandled error from nib_file_read:No such file or directory
init: Caught abort, core dumped

Please advise as soon as possible. Thanks.

[Mattias Backman (mabac)]

On Mon, Dec 19, 2011 at 5:54 PM, Mike Carifio <email address hidden> wrote:
> Public bug reported:
>
> On oneiric i386, I'm building an image file with the following command:
>
> linaro-media-create  --image-file build.img   --hwpack-force-yes --dev
> mx53loco   --binary rootfs.tar.gz  --hwpack hwpack.tar.gz
>
> [If the rootfs and hwpack are needed for debugging, I can provide them.]
>
> I'm using linaro-image-tools '2011.12-0ubuntu1~linaro1'.
>
>
> linaro-media-create consistently gets to the "Populating ... this will take a few minutes ..." message and then hangs my laptop. In /var/log/syslog, I consistently see this message:
>
> init: /tmp/tmp<random>/binary/filesyste.dir/etc/init/plymouth-splash.conf: Unable to reload configuration after override deletion.
> init: /tmp/tmp<random>/binary/filesyste.dir/etc/init: Configuration directory deleted

Since you're not hacking your own version of l-i-t I assume that the
typo in the path was introduced when you pulled it from the log. It
should be "binary/boot/filesystem.dir".

> init: file.c:110: Unhandled error from nib_file_read:No such file or directory
> init: Caught abort, core dumped
> init: file.c:110: Unhandled error from nib_file_read:No such file or directory
> init: Caught abort, core dumped

I can't say that I've ever seen this before. Do you get the same error
if you exchange --image-file for --mmc <device> to write directly to
the card?

>
> Please advise as soon as possible. Thanks.
>
> ** Affects: linaro-image-tools
>     Importance: Undecided
>         Status: New
>
>
> ** Tags: hang oneiric
>
> --
> You received this bug notification because you are subscribed to Linaro
> Image Tools.
> https://bugs.launchpad.net/bugs/906414
>
> Title:
>  linaro-media-create hangs oneiric
>
> Status in Linaro Image Tools:
>  New
>
> Bug description:
>  On oneiric i386, I'm building an image file with the following
>  command:
>
>  linaro-media-create  --image-file build.img   --hwpack-force-yes --dev
>  mx53loco   --binary rootfs.tar.gz  --hwpack hwpack.tar.gz
>
>  [If the rootfs and hwpack are needed for debugging, I can provide
>  them.]
>
>  I'm using linaro-image-tools '2011.12-0ubuntu1~linaro1'.
>
>
>  linaro-media-create consistently gets to the "Populating ... this will take a few minutes ..." message and then hangs my laptop. In /var/log/syslog, I consistently see this message:
>
>  init: /tmp/tmp<random>/binary/filesyste.dir/etc/init/plymouth-splash.conf: Unable to reload configuration after override deletion.
>  init: /tmp/tmp<random>/binary/filesyste.dir/etc/init: Configuration directory deleted
>  init: file.c:110: Unhandled error from nib_file_read:No such file or directory
>  init: Caught abort, core dumped
>  init: file.c:110: Unhandled error from nib_file_read:No such file or directory
>  init: Caught abort, core dumped
>
>  Please advise as soon as possible. Thanks.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-image-tools/+bug/906414/+subscriptions

[Mike Carifio (carifio)]

Yes, same error. --image-file make no difference.

[Mike Carifio (carifio)]

command:

linaro-media-create --image-file try.img --hwpack-force-yes --dev mx53loco --binary <binary_file> --hwpack <hwpack_file>

I've attached the two files, binary and hwpack.

[Guilherme Salgado (salgado)]

Ok, so I was able to reproduce this and although I haven't found out what exactly is causing the system hang, I think I've found some clues. Right before the hang, I see the following process running:

   5959 ? Ss 0:00 /usr/bin/qemu-arm-static /bin/dbus-daemon --system --fork --activation=upstart

With the arm chroot as cwd, obviously. I suspect this process is started by dubs' postint script and is probably related to the hang because the crash comes from upstart (http://paste.ubuntu.com/793055/) when a config directory it is watching (etc/init/ within the arm chroot, in this case) is removed (this happens when we mv the contents of the chroot to the image file).

My first guess was that the process above is what's causing upstart to watch etc/init/ within the chroot but I've killed it before moving the chroot contents to the image file and the system hang just the same. It'd be nice to get somebody who knows something about upstart/dbus to confirm that my hypothesis above makes sense and, if so, maybe they'll know what we can do to avoid it.

One thing that is pretty clear to me, though, is that this issue is specific to this hwpack and is most likely caused by the installation of one or more packages within the arm chroot. We need to find what are those package(s) and see if we can change them so that they can be installed on a chroot without affecting the host.

[Guilherme Salgado (salgado)]

Just noticed that if I kill the "/usr/bin/qemu-arm-static /bin/dbus-daemon" *and* restart dbus "service dbus restart" before moving the chroot contents to the image file, the image creation succeeds without hanging the system.

[Mike Carifio (carifio)]

When you say "change the [packages] so that they can be installed on a chroot without affecting the host", is that for debugging purposes or as a more permanent solution? Hopefully the permanent solution does not involve making specific arm modifications to packages to satisfy linaro-media-create. But let's not get ahead of ourselves, I guess.

[Guilherme Salgado (salgado)]

No, that's not for debugging, and I understand your concern about
changing the package, but I don't think that's something we'd be doing
to satisfy l-m-c. From my observations, I'm pretty sure we'd get a
similar hang by just creating a chroot (same arch, even) with
debootstrap, chrooting into it, installing dbus (or whatever package is
actually affecting the host when installed on the chroot) and removing
the chroot right afterwards. (This is basically what l-m-c is doing)

I don't know how we'd go about fixing this so that l-m-c doesn't cause
your system to hang, but one assumption we make in l-m-c is that we can
safely install packages in the chroot, move it to the sdcard/image-file
and be done with it. If that's not true for all packages, then we might
need to change l-m-c to treat some packages differently so that they
don't affect the host when installed in the chroot (doesn't sound like a
good idea to do that in lmc, and I'm not even sure that'd be possible).

The other alternative I see, which is the one I alluded to in the
previous comment, is to change the packages so that they don't affect
the host system when they're installed in the chroot. As I said, I
understand your concern about doing that, but if we indeed can reproduce
this hang with the steps I listed on the first paragraph, it sounds to
me like a reasonable thing to do.

[Guilherme Salgado (salgado)]

I've just confirmed that installing the dbus package on a arm chroot (using qemu-arm-static) and then removing the chroot right afterwards causes the hang we're seeing here. The same doesn't seem to happen on a non-cross chroot; although the installation of dbus on the chroot fires the dbus-daemon process, it doesn't hang the system when the chroot is removed. This is what I get on a non-cross chroot:

Jan 5 10:28:46 ubuntu kernel: [ 715.398249] init: /home/salgado/chroots/test/etc/init: Configuration directory deleted
Jan 5 10:28:46 ubuntu kernel: [ 715.399442] init: dbus main process (14011) killed by ABRT signal
Jan 5 10:28:46 ubuntu kernel: [ 715.399461] init: dbus main process ended, respawning
Jan 5 10:28:46 ubuntu kernel: [ 715.401067] init: Failed to spawn dbus pre-start process: unable to execute: No such file or directory

[Guilherme Salgado (salgado)]

I've also tried to reproduce the hang by installing dbus using a nano tarball (linaro-o-nano-tar-20111031-0.tar.gz, which I had here already) as the rootfs, but it didn't happen -- got the same error I got on the non-cross chroot. Then I noticed that the two rootfss where I didn't get the hang were either oneirc- or precise-based and the one where we can reproduce the hang is natty-based, which kind of suggests to me that this is a bug on something like upstart or libnih1*, and it's been fixed already.

[Guilherme Salgado (salgado)]

One thing to consider is why the hwpack attached here includes a dbus package when the same version of it is already installed in the (also attached) rootfs? If the hwpack didn't include the dbus package we wouldn't be seeing this hang.

[Milo Casagrande (milo)]

Due to the age of this issue, we are acknowledging that this issue will likely not be fixed, is no longer applicable, or was already fixed by an indirect change. If this issue is still important, please add details and reopen the issue.