xdeb fails to compile kernel

[Amit Kucheria]

Binary package hint: xdeb

I was trying to cross-compile the kernel using the command 'xdeb -aarmel --only-explicit --apt-source linux-linaro-omap'. I am not entirely sure where the problem lies - toolchain, xdeb or the kernel packaging. Filing this so it doesn't get lost.

Toolchain version was: arm-linux-gnueabi-gcc -v
Using built-in specs.
COLLECT_GCC=arm-linux-gnueabi-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
Target: arm-linux-gnueabi
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.5.2-1ubuntu6' --with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.5 --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/arm-linux-gnueabi/include/c++/4.5.2 --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --program-prefix=arm-linux-gnueabi- --includedir=/usr/arm-linux-gnueabi/include --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=arm-linux-gnueabi --with-headers=/usr/arm-linux-gnueabi/include --with-libs=/usr/arm-linux-gnueabi/lib
Thread model: posix
gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-1ubuntu6)

Here is the relevant part of the log:

dh_strip -plinux-headers-2.6.37-1002-linaro-omap
arm-linux-gnueabi-strip:debian/linux-headers-2.6.37-1002-linaro-omap/usr/src/linux-headers-2.6.37-1002-linaro-omap/scripts/kallsyms: File format not recognized
dh_strip: arm-linux-gnueabi-strip --remove-section=.comment --remove-section=.note debian/linux-headers-2.6.37-1002-linaro-omap/usr/src/linux-headers-2.6.37-1002-linaro-omap/scripts/kallsyms returned exit code 1
make: *** [binary-linaro-omap] Error 9
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
debuild: fatal error at line 1335:
dpkg-buildpackage -rfakeroot -d -us -uc -aarmel -b -tc failed
Traceback (most recent call last):
  File "/usr/bin/xdeb", line 849, in <module>
    main()
  File "/usr/bin/xdeb", line 841, in main
    build(options, src, all_srcs[src])
  File "/usr/bin/xdeb", line 596, in build
    spawn(buildpackage, cwd=srcdir)
  File "/usr/lib/xdeb/utils.py", line 29, in spawn
    raise SubprocessException, ret
utils.SubprocessException: 29

[Loïc Minier (lool)]

On my amd64 system, linux-headers-2.6.38-2-generic is arch: amd64 and ships a /usr/src/linux-headers-2.6.38-2-generic/scripts/kallsyms file which is for x86-64.

When you're cross-building the kernel, there are two questions:
a) does kallsyms need to be called during the build; if yes, then you will need a version built for the buildd's architecture
b) do you need to cross-build kallsyms to ship in the target package, or is it already provided in another package/built every time it's needed

Given the output you quote, it seems stripping of debian/linux-headers-2.6.37-1002-linaro-omap/usr/ fails, which means that a file which is about to be shipped in this package can't be stripped. dh_strip in Ubuntu is normally aware of the need to call the cross-strip to strip for another architecture. So my conclusion is that this file is built for your build architecture instead of armel and should either not be in the packages at all, or should be built for armel.

If a) is true, then it's just a matter of installing the right file in the package.

If per b) you actually don't need to ship the file, just stop installing it in the package and your problem is solved too.

In both cases, this points at a problem in the kernel packaging or cross-build rules, but if you run "file debian/linux-headers-2.6.37-1002-linaro-omap/usr/src/linux-headers-2.6.37-1002-linaro-omap/scripts/kallsyms", this should confirm whether the file is indeed of the buildd's architecture as I suspect.

I think John Rigby saw similar issues.

Thanks,

[John Rigby (jcrigby)]

When cross building I run the following script first. It 1. enables ccache which is usefull if you are repeating the build, 2. turns off the tools build (which is only on for the mx51 flavour since the tools are flavour independent so we only build once). 3. disables the dh_strip which is the problem seen here.

#!/bin/sh
## enable ccache
sed -i -e 's/CC.*=.*CROSS_COMPILE.gcc/CC = ccache $(CROSS_COMPILE)gcc/' Makefile
## turn off tools
sed -i -e 's/do_tools.*=.*/do_tools = false/' debian.linaro/rules.d/armel.mk
## turn off dh_strip
sed -i -e 's/dh_strip.*pkghdr.*/echo skipping dh_strip for cross build/' debian/rules.d/2-binary-arch.mk

As Loic mentions, the proper fix for the dh_strip problem is to make the linux-headers properly cross compile so there will be an arm binary there for the arm dh_strip to work on. I have researched but not yet implement that fix. It involves teaching the kernel make that some of the binaries in scripts need to be compiled for both the host and the target.

Also the do_tools issue is fixed (I think) if you install the necessary arm dev packages on your host:
sudo xapt -a armel install libdw-dev
sudo xapt -a armel install libelf-dev

(after adding the right entry to your apt-whatever file...)

[Amit Kucheria (amitk)]

On 11 Feb 09, John Rigby wrote:
> When cross building I run the following script first. It 1. enables
> ccache which is usefull if you are repeating the build, 2. turns off the
> tools build (which is only on for the mx51 flavour since the tools are
> flavour independent so we only build once). 3. disables the dh_strip
> which is the problem seen here.

I don't agree with 2). It would be nice to have the perf tools.

'apt-get install linux-tools' gives me the tools for the 38-2 kernel from the
kernel team. So that doesn't work.

> #!/bin/sh
> ## enable ccache
> sed -i -e 's/CC.*=.*CROSS_COMPILE.gcc/CC = ccache $(CROSS_COMPILE)gcc/' Makefile
> ## turn off tools
> sed -i -e 's/do_tools.*=.*/do_tools = false/' debian.linaro/rules.d/armel.mk
> ## turn off dh_strip
> sed -i -e 's/dh_strip.*pkghdr.*/echo skipping dh_strip for cross build/' debian/rules.d/2-binary-arch.mk

Why can't ccache be an environment variable like CROSS_COMPILE? If set, we use it.

> As Loic mentions, the proper fix for the dh_strip problem is to make the
> linux-headers properly cross compile so there will be an arm binary
> there for the arm dh_strip to work on. I have researched but not yet
> implement that fix. It involves teaching the kernel make that some of
> the binaries in scripts need to be compiled for both the host and the
> target.

the kallsyms is indeed a amd64 binary on my system.

> Also the do_tools issue is fixed (I think) if you install the necessary arm dev packages on your host:
> sudo xapt -a armel install libdw-dev
> sudo xapt -a armel install libelf-dev
>
> (after adding the right entry to your apt-whatever file...)

What should I add to apt sources.list?

I don't believe all this is documented. We need a wiki page for developers to
cross-compile their own kernels (similar to you page to create a new
flavour). People are reverting to their own kernels and processes because it
is too hard to flash a new kernel using the 'ubuntu-way'.

[Loïc Minier (lool)]

> Why can't ccache be an environment variable like CROSS_COMPILE? If set, we use it.

I think this is because the usual way to use ccache is to have a symlink to ccache with the name of your compiler under /usr/lib/ccache, and prepend that dir to PATH, but there is no such link for the cross-toolchain. This is bug #605189; let's not get distracted here.

> the kallsyms is indeed a amd64 binary on my system.

Ok; that's wrong then. Is this binary needed in the package at all? (On any architecture)
If not, we should fix the kernel packaging or upstream rules to not install it.

if it's needed in the package, we should fix the build to produce an ARM one and install that one (maybe the build needs to produce a x86 one as well for the time of the build).

[Amit Kucheria (amitk)]

This seems like a kernel packaging issue, not xdeb. kallsyms on my system is an amd64 binary. It needs to be armel when cross-building.

[Jeremy Foshee (jeremyfoshee)]

Hi Amit,

Please be sure to confirm this issue exists with the latest development release of Ubuntu. ISO CD images are available from http://cdimage.ubuntu.com/daily/current/ . If the issue remains, please run the following command from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux 715429

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

    [This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

[John Rigby (jcrigby)]

linux-headers cross compiles correctly in the next release of linux-linaro-*.

There are two remaining problems. for the first I have added a bug:

Cross compiling requires turning off do_tools: https://bugs.launchpad.net/ubuntu/+source/linux-linaro-mx51/+bug/737600

The second issue is that building the kernel package is hard for anyone not accustomed to ubuntu kernel packaging. I have started a wiki page but this still a WIP.