ARM kernel builds fail with GCC 11

[Juerg Haefliger]

While this might be a kernel build problem (see LP: #1939308) I wanted to raise the issue against the toolchain just so we can make sure it's *not* a toolchain bug. Basically the following behavior changed from gcc 10:

$ arm-linux-gnueabihf-gcc-10 -march=armv7-a -c -x c /dev/null ; echo $?
0

$ arm-linux-gnueabihf-gcc-11 -march=armv7-a -c -x c /dev/null ; echo $?
cc1: error: ‘-mfloat-abi=hard’: selected architecture lacks an FPU
1

Presumably the GCC 11 hard-float compiler now enforces the default config --with-float=hard?

[dann frazier (dannf)]

I wonder if this is due to:

gcc-11 (11.1.0-2) experimental; urgency=medium

  [...]
  * For armhf configure --with-arch=<arch>+fp, dropping the --with-fpu= option.
  [...]

FWIW, I'm seeing something similar w/ edk2.

[dann frazier (dannf)]

Yeah, I think what's going on is that - due to the above change - gcc-11 no longer enables an FPU independently of the selected arch. The FPU is only enabled because the default -march is armv7-a+fp. The kernel build is overriding GCC's (new) built-in default march to one that does not have "+fp", so no flags remain to enable the FPU. edk2 does something similar, and changing it's forced -march from armv7-a to armv7-a+fp works for me.

[Michael Hudson-Doyle (mwhudson)]

Oh, I've seen this in some other packages too (boost1.74 was one I think). Do we need to change dpkg to have different default CFLAGS or something?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[dann frazier (dannf)]

Interesting - here's what I see in the boost1.74 log:

    "g++" -x assembler-with-cpp -fPIC -pthread -O3 -finline-functions -Wno-inline -Wall -g -fvisibility=hidden -Wdate-time -D_FORTIFY_SOURCE=2 -DBOOST_ALL_NO_LIB=1 -DBOOST_CONTEXT_DYN_LINK=1 -DBOOST_CONTEXT_SOURCE -DBOOST_DISABLE_ASSERTS -DNDEBUG -I"." -c -o "bin.v2/libs/context/build/gcc-11/release/debug-symbols-on/threading-multi/visibility-hidden/asm/jump_arm_aapcs_elf_gas.o" "libs/context/src/asm/jump_arm_aapcs_elf_gas.S"

jump_arm_aapcs_elf_gas.S: Assembler messages:
jump_arm_aapcs_elf_gas.S:57: Error: selected processor does not support `vstmia sp,{d8-d15}' in ARM mode
jump_arm_aapcs_elf_gas.S:68: Error: selected processor does not support `vldmia sp,{d8-d15}' in ARM mode

It's not messing with -march, so seems like g++ should default to having an FPU?

[Matthias Klose (doko)]

yes, that's what is supposed to do, but apparently it's not doing that. Now configured with --with-fpu=vfpv3-d16 again, which lets the boost build at least succeed again.

Still need to figure out if we need to re-upload packages built after the gcc-defaults change and before the gcc fix.

[Juerg Haefliger (juergh)]

I wonder if this is related but cpio on impish arm64 seems broken. All my kernel builds on impish fail when cpio is invoked. In a hirsute arm64 chroot:

(hirsute-arm64)ubuntu@seidel:~$ find foo/ -print | cpio -pd --preserve-modification-time $(pwd)/0000000000111111111122222222223333333333444444444455555555556666666666777777777788888888889999999999
22511 blocks
(hirsute-arm64)ubuntu@seidel:~$ pull-lp-debs cpio impish
Found cpio 2.13+dfsg-4ubuntu1 in impish
Downloading cpio_2.13+dfsg-4ubuntu1_arm64.deb from ports.ubuntu.com (0.078 MiB)
Downloading cpio-win32_2.13+dfsg-4ubuntu1_all.deb from launchpadlibrarian.net (0.077 MiB)
$ sudo dpkg -i cpio_2.13+dfsg-4ubuntu1_arm64.deb
(Reading database ... 15849 files and directories currently installed.)
Preparing to unpack cpio_2.13+dfsg-4ubuntu1_arm64.deb ...
Unpacking cpio (2.13+dfsg-4ubuntu1) over (2.13+dfsg-4) ...
Setting up cpio (2.13+dfsg-4ubuntu1) ...
(hirsute-arm64)ubuntu@seidel:~$ rm -rf 0000000000111111111122222222223333333333444444444455555555556666666666777777777788888888889999999999/
(hirsute-arm64)ubuntu@seidel:~$ find foo/ -print | cpio -pd --preserve-modification-time $(pwd)/0000000000111111111122222222223333333333444444444455555555556666666666777777777788888888889999999999
cpio: e: Cannot stat: No such file or directory
cpio: oe/autoload.sh: Cannot stat: No such file or directory
cpio: e-analyzer/Makefile: Cannot stat: No such file or directory
cpio: i/mvm/Makefile: Cannot stat: No such file or directory
cpio: ile: Cannot stat: No such file or directory
cpio: /Kconfig: Cannot stat: No such file or directory
cpio: stmem: Cannot stat: No such file or directory
cpio: blank line ignored
cpio: blank line ignored
cpio: h: Cannot stat: No such file or directory
cpio: e-build.sh: Cannot stat: No such file or directory
cpio: /forwarding/ip6gre_inner_v4_multipath.sh: Cannot stat: No such file or directory
cpio: ers/net/mlxsw/router_scale.sh: Cannot stat: No such file or directory
22470 blocks

That's exactly the type of error I'm seeing when trying to build a kernel in an impish chroot. cpio seems to choke on the long directory name:

(hirsute-arm64)ubuntu@seidel:~$ rm -rf 0000000000111111111122222222223333333333444444444455555555556666666666777777777788888888889999999999/
(hirsute-arm64)ubuntu@seidel:~$ find foo/ -print | cpio -pd --preserve-modification-time $(pwd)/00000000001111111111222222222233333333334444444444
22511 blocks

[dann frazier (dannf)]

On Fri, Aug 13, 2021 at 7:35 AM Juerg Haefliger
<email address hidden> wrote:
>
> I wonder if this is related but cpio on impish arm64 seems broken.

I wouldn't think so - this change should only impact armhf.

  -dann

[Michael Hudson-Doyle (mwhudson)]

Juerg, that fun is from bug 1939973

[Juerg Haefliger (juergh)]

We're also seeing dkms build failures on armhf due to this: https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/armhf/a/adv-17v35x/20210825_233224_cd9ac@/log.gz

[Matthias Klose (doko)]

this log shows:

make: Entering directory '/usr/src/linux-headers-5.13.0-14-generic'
  CC [M] /var/lib/dkms/adv-17v35x/5.0.6.0-1/build/adv17v35x.o

please could you provide a verbose build, show the exact invocation of the compiler?

[Matthias Klose (doko)]

> $ arm-linux-gnueabihf-gcc-11 -march=armv7-a -c -x c /dev/null
> cc1: error: ‘-mfloat-abi=hard’: selected architecture lacks an FPU

you have to use -march=armv7-a+fp for now on.

[Dimitri John Ledkov (xnox)]

@juerg

From your autopkgtest result:

Building module:
cleaning build area...
make -j4 KERNELRELEASE=5.13.0-1006-raspi -C /lib/modules/5.13.0-1006-raspi/build M=/var/lib/dkms/adv-17v35x/5.0.6.0-1/build....
cleaning build area...

DKMS: build completed.

Building module:
cleaning build area...
make -j4 KERNELRELEASE=5.13.0-14-generic -C /lib/modules/5.13.0-14-generic/build M=/var/lib/dkms/adv-17v35x/5.0.6.0-1/build....(bad exit status: 2)
Error! Bad return status for module build on kernel: 5.13.0-14-generic (armv7l)

Because the toolchain fix was only applied in the linux-raspi flavour so far
, and not generic.

I.e. gcc-11 did not migrate, and "UBUNTU: SAUCE: arm: Fix instruction set selection for GCC 11" is not in generic kernel, yet.

so it is expected that dkms builds for generic kernel fail on armhf with gcc-11 from impish-release.

[Sergio Durigan Junior (sergiodj)]

I've added php8.1 as a task because this bug is affecting the package as well.

FWIW, I wasn't able to workaround this problem by just using -march=armv7-a+fp, even though I agree it should have worked. I had to actually pass -mfpu=vfpv3-d16 (no need to pass -march).

Either way, I just wanted to let you folks know that this is affecting other packages too.

[Matthias Klose (doko)]

$ arm-linux-gnueabihf-gcc-11 -march=armv7-a+fp -c -x c /dev/null ; echo $?
0

that works, contrary what is mentioned in comment #14

[Sergio Durigan Junior (sergiodj)]

On Wednesday, November 10 2021, Matthias Klose wrote:

> $ arm-linux-gnueabihf-gcc-11 -march=armv7-a+fp -c -x c /dev/null ; echo $?
> 0
>
> that works, contrary what is mentioned in comment #14

I don't understand your statement here. I never claimed that gcc-11
"doesn't work". I never claimed that using -march=armv7-a+fp "never
works". I claimed that using this flag to build php8.1 doesn't seem to
work.

You can look at the build logs here:

  https://launchpad.net/~sergiodj/+archive/ubuntu/php8.1-armhf/+packages
  https://launchpadlibrarian.net/568231957/buildlog_ubuntu-jammy-armhf.php8.1_8.1.0~rc4-1ubuntu3~ppa1_BUILDING.txt.gz

This is the debdiff used in the PPA above:

  https://launchpadlibrarian.net/568229387/php8.1_8.1.0~rc4-1ubuntu2_8.1.0~rc4-1ubuntu3~ppa1.diff.gz

diff -Nru php8.1-8.1.0~rc4/debian/rules php8.1-8.1.0~rc4/debian/rules
--- php8.1-8.1.0~rc4/debian/rules 2021-10-22 23:59:56.000000000 +0000
+++ php8.1-8.1.0~rc4/debian/rules 2021-11-10 17:49:57.000000000 +0000
@@ -146,7 +146,7 @@
 endif

 ifeq (arm-linux-gnueabihf,$(DEB_HOST_GNU_TYPE))
- DEB_CFLAGS_MAINT_APPEND += -mfpu=vfpv3-d16
+ DEB_CFLAGS_MAINT_APPEND += -march=armv7-a+fp
 endif

 # Enable producing of debugging information

The error we have is this:

jump_arm_aapcs_elf_gas.S: Assembler messages:
jump_arm_aapcs_elf_gas.S:57: Error: selected processor does not support `vstmia sp,{d8-d15}' in ARM mode
jump_arm_aapcs_elf_gas.S:68: Error: selected processor does not support `vldmia sp,{d8-d15}' in ARM mode

Note that using "-mfpu=vfpv3-d16" does work, and is what php8.1 uses
right now in order to properly build in Ubuntu.

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

[Ondřej Surý (ondrej)]

php-swoole on armhf impish is also affected: https://launchpad.net/~ondrej/+archive/ubuntu/php/+build/22393235

[dann frazier (dannf)]

hm.. yeah, I can reproduce:

dannf@xps13:/tmp/php8.1-8.1.0~rc4$ arm-linux-gnueabihf-gcc -IZend/asm/ "-I/tmp/php8.1-8.1.0~rc4/Zend/asm/" "-I/tmp/php8.1-8.1.0~rc4/ext-build/include" "-I/tmp/php8.1-8.1.0~rc4/ext-build/main" "-I/tmp/php8.1-8.1.0~rc4" "-I/tmp/php8.1-8.1.0~rc4/ext-build/ext/date/lib" "-I/tmp/php8.1-8.1.0~rc4/ext/date/lib" -I/usr/include/libxml2 -I/usr/include/arm-linux-gnueabihf -I/usr/include/enchant-2 -I/usr/include/glib-2.0 -I/usr/lib/arm-linux-gnueabihf/glib-2.0/include -I/usr/include/c-client "-I/tmp/php8.1-8.1.0~rc4/ext/mbstring/libmbfl" "-I/tmp/php8.1-8.1.0~rc4/ext-build/ext/mbstring/libmbfl" "-I/tmp/php8.1-8.1.0~rc4/ext/mbstring/libmbfl/mbfl" "-I/tmp/php8.1-8.1.0~rc4/ext-build/ext/mbstring/libmbfl/mbfl" -I/usr/include/postgresql -I/usr/include/pspell -I/usr/include/editline -I/usr/include/tidy "-I/tmp/php8.1-8.1.0~rc4/ext-build/TSRM" "-I/tmp/php8.1-8.1.0~rc4/ext-build/Zend" "-I/tmp/php8.1-8.1.0~rc4/main" "-I/tmp/php8.1-8.1.0~rc4/Zend" "-I/tmp/php8.1-8.1.0~rc4/TSRM" "-I/tmp/php8.1-8.1.0~rc4/ext-build/" -Wdate-time -D_FORTIFY_SOURCE=2 -fno-common -Wstrict-prototypes -Wformat-truncation -Wlogical-op -Wduplicated-cond -Wno-clobbered -Wall -Wextra -Wno-strict-aliasing -Wno-unused-parameter -Wno-sign-compare -g -O2 "-ffile-prefix-map=/tmp/php8.1-8.1.0~rc4=." -fstack-protector-strong -Wformat -Werror=format-security -O2 -Wall -pedantic -fsigned-char -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -march=armv7-a+fp -g -fvisibility=hidden -Wimplicit-fallthrough=1 -DZEND_SIGNALS -c "/tmp/php8.1-8.1.0~rc4/Zend/asm/jump_arm_aapcs_elf_gas.S" -MMD -MF Zend/asm/jump_arm_aapcs_elf_gas.dep -MT Zend/asm/jump_arm_aapcs_elf_gas.lo -fPIC -DPIC -o Zend/asm/.libs/jump_arm_aapcs_elf_gas.o
jump_arm_aapcs_elf_gas.S: Assembler messages:
jump_arm_aapcs_elf_gas.S:57: Error: selected processor does not support `vstmia sp,{d8-d15}' in ARM mode
jump_arm_aapcs_elf_gas.S:68: Error: selected processor does not support `vldmia sp,{d8-d15}' in ARM mode

Is it perhaps because the -march arg isn't be passed to the assembler? If I add -Wa,-march=armv7-a+fp:

dannf@xps13:/tmp/php8.1-8.1.0~rc4$ arm-linux-gnueabihf-gcc -IZend/asm/ "-I/tmp/php8.1-8.1.0~rc4/Zend/asm/" "-I/tmp/php8.1-8.1.0~rc4/ext-build/include" "-I/tmp/php8.1-8.1.0~rc4/ext-build/main" "-I/tmp/php8.1-8.1.0~rc4" "-I/tmp/php8.1-8.1.0~rc4/ext-build/ext/date/lib" "-I/tmp/php8.1-8.1.0~rc4/ext/date/lib" -I/usr/include/libxml2 -I/usr/include/arm-linux-gnueabihf -I/usr/include/enchant-2 -I/usr/include/glib-2.0 -I/usr/lib/arm-linux-gnueabihf/glib-2.0/include -I/usr/include/c-client "-I/tmp/php8.1-8.1.0~rc4/ext/mbstring/libmbfl" "-I/tmp/php8.1-8.1.0~rc4/ext-build/ext/mbstring/libmbfl" "-I/tmp/php8.1-8.1.0~rc4/ext/mbstring/libmbfl/mbfl" "-I/tmp/php8.1-8.1.0~rc4/ext-build/ext/mbstring/libmbfl/mbfl" -I/usr/include/postgresql -I/usr/include/pspell -I/usr/include/editline -I/usr/include/tidy "-I/tmp/php8.1-8.1.0~rc4/ext-build/TSRM" "-I/tmp/php8.1-8.1.0~rc4/ext-build/Zend" "-I/tmp/php8.1-8.1.0~rc4/main" "-I/tmp/php8.1-8.1.0~rc4/Zend" "-I/tmp/php8.1-8.1.0~rc4/TSRM" "-I/tmp/php8.1-8.1.0~rc4/ext-build/" -Wdate-time -D_FORTIFY_SOURCE=2 -fno-common -Wstrict-prototypes -Wformat-truncation -Wlogical-op -Wdu...

[Michael Hudson-Doyle (mwhudson)]

FWIW I fixed a similar issue in supertuxkart recently by patching the .S file to have a ".fpu vfp" directive in it -- although probably ".arch armv7-a+fp" would have been more appropriate.

It does seem strange that "gcc -march foo bar.s" does not pass -march to the assembler but I have no real idea if that expectation is grounded in reality -- when compiling C, the value of march is passed to the assembler via a directive so maybe "march" is really only an argument to gcc when gcc is doing the compiling and not when it is being used as a frontend to the assembler.

[Michael Hudson-Doyle (mwhudson)]

Here's some discussion from over a decade ago on the topic https://gcc.gnu.org/pipermail/gcc-patches/2009-May/262656.html (spoiler: it doesn't really go anywhere)

[Sergio Durigan Junior (sergiodj)]

This may very well be the issue. Thanks for the reply, dann.

I'm running a test build using -Wa,-march... to check if the build succeeds now.