run-init on omap3, omap4 in natty dies if busybox is built with -marm

[Oliver Grawert]

trying to boot a natty image on the omap4 platform works fine for teh first part where jasper-initramfs resizes, configures and reboots teh system. as soon as the reboot happened and the rootfs is trying to be mounted the boot dies with:

/init: exec: line 331: run-init: Unknown error 17718852

[Oliver Grawert (ogra)]

seems linaro has a similar issue in bug 683582

[Thorsten Glaser (mirabilos)]

This looks similar to Debian #334917

[Paul Larson (pwlars)]

Also, see Linaro bug #683217.

Bug #683582 in Linaro actually had two bugs in it. Since that issue, and the one mentioned here, seem to be a duplicate of Bug the one I mentioned above, I changed it to focus on just the linaro-media-create bug.

[Paul Larson (pwlars)]

I'll dup mine to here since there seems to be more activity here already.

[Oliver Grawert (ogra)]

researching the archive, it seems that klibc did not change since maverick (not even a rebuild with the new toolchain). initramfs-tools only had a minor change as well.

i was suspecting something with the handed over args to be wrong. passing init=/bin/bash shows the same behavior, running run-init from an initramfs shell manually with teh right args also shows the same results.

[Thorsten Glaser (mirabilos)]

Looking at the error and the m68k bug noted, and remembering some planet
postings about armhf (Debian) and *buntu’s ARM related things, I can only wonder
if the same is true about klibc not knowing the correct (syscall? procedure call?) ABI.

[Tom Gall (tom-gall)]

Using linaro's natty daily snapshot for headless dated 1201 and the hwpack for omap3 dated 1201 and using a linaro-media create that was branched on 11-23 (but with no local changes) my beagle board C4 boots. tty doesn't work but serial cons is fine:

here's the boot log.

http://pastebin.ubuntu.com/538741/

[Tom Gall (tom-gall)]

I've now been able to replicate this on my beagle xm (but not my C4) same headless 1201 natty build, same linaro-media create, same hwpack.

Boot log:
http://pastebin.ubuntu.com/538759/

[Michael Casadevall (mcasadevall)]

Debugging info on what run-init is being called with:
13:58:17 < NCommander> ogra_ac: rootmnt is "/root"
13:58:22 < NCommander> init is "/sbin/init"
13:58:26 < NCommander> arguments: "fixrtc"

Arguments to run-init are sane and correct.

[Jamie Bennett (jamiebennett)]

Seems we have a successful boot on the C4 and a failed boot with the *same* sd card with the XM.

[Tom Gall (tom-gall)]

That was me ... I have an microSD / SD adapter, using the netbook-efl image, booted on the C4 ... moved the microSD card over to the Beagle Xm failed as noted above in this bug

[Tom Gall (tom-gall)]

so using some echo magic sprinkles in the initrd this seems to be coming from the init script itself ... and the line 331 in the error message in line 331 in init which is the following command :

exec run-init ${rootmnt} ${init} "$@" <${rootmnt}/dev/console >${rootmnt}/dev/co
nsole 2>&1

From my debug output on the xm it seems to be trying to run the following:

+ exec run-init /root /sbin/init rootwait ro earlyprintk fixrtc nocompcache

full output here:

http://pastebin.ubuntu.com/538841/

[Tom Gall (tom-gall)]

Now running with the same SD card on my beagle C4 which boots to a login ... I get no debug output at all ... it's like we're never running init out of the uInitrd ...

anyway out of time today.

[John Rigby (jcrigby)]

If I replace the /bin/sh in the broken natty initrd with one from a maverick final the problem goes away. The difference is the busybox-initramfs package, natty is 1:1.17.1-7ubuntu2 maverick is 1:1.15.3-1ubuntu5. No other changes were made.

[John Rigby (jcrigby)]

There is a .37 kernel package for omap in the linaro maintainers kernel ppa if anyone is inclined to try it. There is also a meta package there. Beware that the omap4 support is still a work in progress. The main missing feature is framebuffer.

ppa:linaro-maintainers/kernel

[Oliver Grawert (ogra)]

1.17.1-6ubuntu1 is the first package exposing the problem

[Oliver Grawert (ogra)]

rebuilding nattys busybox under mavericks toolchain (and dropping -marm) makes it work fine

[Oliver Grawert (ogra)]

doing the same as above with the natty toolchain works fine as well, so it seems the -marm gets in our way.

[Launchpad Janitor (janitor)]

This bug was fixed in the package busybox - 1:1.17.1-7ubuntu3

---------------
busybox (1:1.17.1-7ubuntu3) natty; urgency=low

  * drop -marm from armel build options to make run-init work again
  (LP: #683683)
 -- Oliver Grawert <email address hidden> Thu, 02 Dec 2010 15:09:33 +0100

[Dr. David Alan Gilbert (davidgil-uk)]

Added gcc since you wouldn't have thought -marm would break something like that.

[Matthias Klose (doko)]

gcc-defaults is wrong, unless proven that this occurs with all gcc versions.

[Dave Martin (dave-martin-arm)]

klibc has some outstanding ARM/Thumb interworking issues ...
can someone please apply the patch from this bug:

https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/527720 (thumb2 porting issues identified: klibc uses mov.*pc)

... and see if it makes any difference?

[Michael Hope (michaelh1)]

I'm a bit of a initrd newbie so I wrote up the steps in testing under QEMU here:
 https://wiki.linaro.org/MichaelHope/Sandbox/DebuggingOnInitrd

[Jamie Bennett (jamiebennett)]

On 6 Dec 2010, at 03:33, Michael Hope <email address hidden> wrote:

> I'm a bit of a initrd newbie so I wrote up the steps in testing under QEMU here:
> https://wiki.linaro.org/MichaelHope/Sandbox/DebuggingOnInitrd

Great resource Michael, thanks for putting it together.

[Oliver Grawert (ogra)]

@dave, could you open a new bug for klibc ? this bug is not klibc related. busybox-initramfs links against glibc and given that busybox' exec behavior changes if we use -marm or not this one is rather a toolchain one.

[Oliver Grawert (ogra)]

@michael,if you only want to debug a running initrd you dont need to repack it, just boot with the break= option (some of the valid values are: top, premount, bottom).
that way you can get a shell inside the initrd, do your debugging tasks and go on booting with ctrl-D
note though that in case of run-init this wont help a lot, since run-init can only be started by pid 1 (break spawns a subshell with new pid)

[Dave Martin (dave-martin-arm)]

@ogra, there is a bug open on klibc -- the one I referenced above: https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/527720

Does it need to be prodded?

[Dave Martin (dave-martin-arm)]

@michael - rdinit=/bin/sh is highly use for initramfs debugging too if you're not aware

[Oliver Grawert (ogra)]

@dave, ah, thanks, yeah, we should retarget it if that patch is complete now (note that this bug is just exposing a busybox issue, the run-init error is apparently just a side effect of exec misbehaving if busybox is built with -marm)

[Dave Martin (dave-martin-arm)]

I don't have the right filesystem in front of me to debug, but this recipe should work for debugging PID 1 with gdb (assuming you have a platform already set up with a populated root filesystem etc.):

Connect a keyboard and monitor before booting.

Boot with rdinit=/bin/sh console=<appropriate serial console device>

mkdir /proc /sys /mnt
mount -ntproc proc /proc
mount -ntsysfs sysfs /sys
mknod -m0666 /dev/null c 1 3
udevd --daemon &
# wait a moment
udevadm trigger
# wait a moment
ps
kill <pid of parent udevd process>
mount -oro -text4 /dev/mmcblk0p2 /mnt
LD_LIBRARY_PATH=/mnt/lib:/mnt/usr/lib:/lib; export LD_LIBRARY_PATH
for x in 9 10 11 12; do /mnt/usr/bin/setsid /mnt/bin/bash </dev/tty$x >/dev/tty$x 2>&1 & done

# now you have some usable ttys
# on one of them, type:
/mnt/usr/bin/gdb
attach 1
tcatch exec
continue

# one the console, type
exec /init

# /bin/sh should get exec'd as PID 1, and gdb will stop it
# you'll probably have to load the relevant debug symbols manually etc.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Rolf Leggewie (r0lf)]

natty has seen the end of its life and is no longer receiving any updates. Marking the natty task for this ticket as "Won't Fix".