Do a sanity check to make sure we can run armel binaries in the chroot

[Tom Gall]

using latest from bzr the following command fails with the following output:

root@neptune:/home/tgall# ~tgall/bzr/linaro-image-tools/linaro-media-create --mmc /dev/sdc --rootfs ext4 --dev panda --hwpack ~tgall/Downloads/hwpack_linaro-panda-x11-base_20110722-0_armel_supported.tar.gz --binary /bld/developer-cross/binary-tar.tar.gz

I see...
Device Mount point Size
/dev/sda none 152627MB
/dev/sda1 / 146899MB
/dev/sda2 none 5726MB
/dev/sda5 none 5726MB
/dev/sdb none 0MB
/dev/sdc none 7695MB
/dev/sdc1 none 51MB
/dev/sdc2 none 7643MB
/dev/sdd none 0MB
/dev/sde none 0MB
/dev/sr0 none 0MB
Are you 100% sure, on selecting [/dev/sdc] (y/n)? y
------------------------------------------------------------
Installing (linaro-hwpack-install) hwpack_linaro-panda-x11-base_20110722-0_armel_supported.tar.gz in target rootfs.
chroot: failed to run command `linaro-hwpack-install': No such file or directory
proc umounted
Traceback (most recent call last):
  File "/home/tgall/bzr/linaro-image-tools/linaro-media-create", line 146, in <module>
    verified_files, *hwpacks)
  File "/home/tgall/bzr/linaro-image-tools/linaro_image_tools/media_create/chroot_utils.py", line 64, in install_hwpacks
    install_hwpack(chroot_dir, hwpack_file, hwpack_force_yes or hwpack_verified)
  File "/home/tgall/bzr/linaro-image-tools/linaro_image_tools/media_create/chroot_utils.py", line 85, in install_hwpack
    cmd_runner.run(args, as_root=True, chroot=chroot_dir).wait()
  File "/home/tgall/bzr/linaro-image-tools/linaro_image_tools/cmd_runner.py", line 100, in wait
    raise SubcommandNonZeroReturnValue(self._my_args, returncode)
linaro_image_tools.cmd_runner.SubcommandNonZeroReturnValue: Sub process "['chroot', '/tmp/tmpVxkBHQ/binary', 'linaro-hwpack-install', '/hwpack_linaro-panda-x11-base_20110722-0_armel_supported.tar.gz']" returned a non-zero value: 127

yet the command in question does exist note:

root@neptune:/home/tgall# ls bzr/linaro-image-tools/linarolinaro-android-media-create linaro-image-indexerlinaro-hwpack-create linaro_image_tools/linaro-hwpack-install linaro-media-create
linaro-hwpack-replace

[Tom Gall (tom-gall)]

output from : strace -f -etrace=process -o lmc.output ./linaro-media-create --mmc /dev/sdc --rootfs ext4 --dev panda --hwpack ~tgall/Downloads/hwpack_linaro-panda-x11-base_20110722-0_armel_supported.tar.gz --binary /bld/developer-cross/binary-tar.tar.gz

[Tom Gall (tom-gall)]

I commented out the TMP_DIR removal code in linaro-media-create, restarted and after it failed cd'd into the entrails.

root@neptune:/tmp/tmpsNSe_7/binary/usr/bin# ls linaro*
ls: cannot access linaro*: No such file or directory

So it would very much appear that linaro-hwpack-install didn't make it inside.

[Guilherme Salgado (salgado)]

On Fri, 2011-07-22 at 16:34 +0000, Tom Gall wrote:
> I commented out the TMP_DIR removal code in linaro-media-create,
> restarted and after it failed cd'd into the entrails.
>
> root@neptune:/tmp/tmpsNSe_7/binary/usr/bin# ls linaro*
> ls: cannot access linaro*: No such file or directory
>
> So it would very much appear that linaro-hwpack-install didn't make it
> inside.

That's probably because we remove l-h-i from the unpacked rootfs before
we copy it over to its final destination. A simpler way to debug this
would be to stick a "import pdb; pdb.set_trace()" after line 77 of
linaro_image_tools/media_create/chroot_utils.py

That way it will stop and only continue when you type c<Enter>

[Tom Gall (tom-gall)]

The rootfs is located on people.linaro.org ~tgall/public/binary-tar.tar.gz

[Tom Gall (tom-gall)]

I tried the tool again with what should be a known good rootfs. linaro-media-create still fails.

root@neptune:/home/tgall/bzr/linaro-image-tools# ./linaro-media-create --mmc /dev/sdc --rootfs ext4 --dev panda --hwpack ~tgall/Downloads/hwpack_linaro-panda-x11-base_20110722-0_armel_supported.tar.gz --binary ~tgall/Downloads/developer-n-tar-20110721-0.tar.gz

I see...
Device Mount point Size
/dev/sda none 152627MB
/dev/sda1 / 146899MB
/dev/sda2 none 5726MB
/dev/sda5 none 5726MB
/dev/sdb none 0MB
/dev/sdc none 7695MB
/dev/sdc1 none 51MB
/dev/sdc2 none 7643MB
/dev/sdd none 0MB
/dev/sde none 0MB
/dev/sr0 none 0MB
Are you 100% sure, on selecting [/dev/sdc] (y/n)? y
------------------------------------------------------------
Installing (linaro-hwpack-install) hwpack_linaro-panda-x11-base_20110722-0_armel_supported.tar.gz in target rootfs.
chroot: failed to run command `linaro-hwpack-install': No such file or directory
proc umounted
Traceback (most recent call last):
  File "./linaro-media-create", line 146, in <module>
    verified_files, *hwpacks)
  File "/home/tgall/bzr/linaro-image-tools/linaro_image_tools/media_create/chroot_utils.py", line 64, in install_hwpacks
    install_hwpack(chroot_dir, hwpack_file, hwpack_force_yes or hwpack_verified)
  File "/home/tgall/bzr/linaro-image-tools/linaro_image_tools/media_create/chroot_utils.py", line 85, in install_hwpack
    cmd_runner.run(args, as_root=True, chroot=chroot_dir).wait()
  File "/home/tgall/bzr/linaro-image-tools/linaro_image_tools/cmd_runner.py", line 100, in wait
    raise SubcommandNonZeroReturnValue(self._my_args, returncode)
linaro_image_tools.cmd_runner.SubcommandNonZeroReturnValue: Sub process "['chroot', '/tmp/tmpod2IyS/binary', 'linaro-hwpack-install', '/hwpack_linaro-panda-x11-base_20110722-0_armel_supported.tar.gz']" returned a non-zero value: 127

[Tom Gall (tom-gall)]

I tried a completely fresh bzr clone lp:linaro-image-tools. Same problem.

[Tom Gall (tom-gall)]

Rebooted and it fixed it which makes me wonder if there isn't some interaction between linaro-media-create and binfmt + qemu-static.

Specifically

modprobe binfmt_misc
mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/exp/qemu-arm-static:' > /proc/sys/fs/binfmt_misc/register

which when qemu-arm-static is copied into /exp in the chroot, allows one to be able to chroot ./chroot somerandomArmProgram to run on arm "automagically".

[Guilherme Salgado (salgado)]

linaro-hwpack-install will run apt-get on the unpacked rootfs, and it
relies on qemu-user-static for that. AFAIK, in Ubuntu the
qemu-user-static package sets things up in /proc/sys/fs/binfmt_misc for
that to work, so this kind of suggests to me that these settings were
somehow screwed in your system and were restored after you rebooted?

[Tom Gall (tom-gall)]

it would seem that someone, somewhere is seting up binfmt_misc + qemu-arm-static to be located /usr/bin. I say this given the following:

root@neptune:/bld/developer-cross# chroot ./chroot /bin/bash --login
chroot: failed to run command `/bin/bash': No such file or directory
root@neptune:/bld/developer-cross# ls chroot/bin/
bash ln sleep
...

root@neptune:/bld/developer-cross# cp chroot/exp/qemu-arm
qemu-armeb-static qemu-arm-static
root@neptune:/bld/developer-cross# cp chroot/exp/qemu-arm* chroot/usr/bin/.
root@neptune:/bld/developer-cross# chroot ./chroot /bin/bash --login
root@neptune:/#

And thus it works. I'd been putting qemu-static* into /exp so I could easily delete it from the chroot on cleanup of the rootfs as it was being built.

I haven't seen a way to query where qemu has to be installed in order for apps of that arch to work via binfmt. I'm happy to change live-build3 so that it uses this same hardcoded position, just be nice to have a way to programmatically do the right thing.

[Tom Gall (tom-gall)]

So in summary by setting echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/exp/qemu-arm-static:' > /proc/sys/fs/binfmt_misc/register

this will break linaro-media-create.

the only valid setting to not break linaro-media-create (and soon to be live-build3 when doing cross builds) is to set the location of qemu-arm-static in /usr/bin.

[Ricardo Salveti (rsalveti)]

Ubuntu sets the interpreter path by default to /usr/bin/qemu-arm-static, so unless you change it back after your tests, I'd recommend you to use it with the default path.

[Rex Tsai (chihchun)]

I encountered same issue, and turned out there is some other program ( scratchbox in my case).

You can check /proc/sys/fs/binfmt_misc to see which binary formats have been registered, and disable them by "echo -1" into these proc files.

Ex: echo -1 > /proc/sys/fs/binfmt_misc/sbox-arm

Just some tips for people who might experienced same problem.

[Guilherme Salgado (salgado)]

One thing we could do here would be to run a test (e.g. 'chroot /path/to/root ls') before we try installing the hwpack. If that test fails we could then tell the user that qemu-user-static is not properly setup or something (scratchbox) messed with its configs.

[Tom Gall (tom-gall)]

Seems quite reasonable. If anything would help with debug in the cases where the default interpreter path isn't being used.