Toolchain sometimes fails to build; seems racy

[Loïc Minier]

Hi,

While rolling RC3, the minimal image would repeatedly fail to build while SDK and LAMP images built fine. Building the same minimal image on a dedicated EC2 vm of a different type (high-memory rather than just a large instance) would pass. Some of these builds were happening on vms running other jobs, others not.

Sample failure:
https://ci.linaro.org/jenkins/view/OpenEmbedded%20Builds/job/openembedded-armv8-minimal/33/console

This seems related to stack protector:
| gcc -c -isystem/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/sysroots/x86_64-linux/usr/include -O2 -pipe -DIN_GCC -DCROSS_DIRECTORY_STRUCTURE -W -Wall -Wwrite-strings -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes -Wmissing-format-attribute -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -Wold-style-definition -Wc++-compat -DHAVE_CONFIG_H -I. -I. -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/. -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/../include -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/../libcpp/include -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/../libdecnumber -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/../libdecnumber/dpd -I../libdecnumber /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/calls.c -o calls.o
| gcc -c -isystem/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/sysroots/x86_64-linux/usr/include -O2 -pipe -DIN_GCC -DCROSS_DIRECTORY_STRUCTURE -W -Wall -Wwrite-strings -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes -Wmissing-format-attribute -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -Wold-style-definition -Wc++-compat -DHAVE_CONFIG_H -I. -I. -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/. -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/../include -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/../libcpp/include -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/../libdecnumber -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/../libdecnumber/dpd -I../libdecnumber /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/cfg.c -o cfg.o
| /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/builtins.c: In function 'expand_builtin_setjmp_setup':
| /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/builtins.c:827:8: warning: implicit declaration of function 'STACK_SAVEAREA_MODE' [-Wimplicit-function-declaration]
| /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/builtins.c:827:8: warning: enum conversion in initialization is invalid in C++ [-Wc++-compat]
| /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/builtins.c: In function 'expand_builtin_longjmp':
| /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/builtins.c:948:8: warning: enum conversion in initialization is invalid in C++ [-Wc++-compat]
| /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/builtins.c: In function 'expand_builtin_nonlocal_goto':
| /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/builtins.c:1051:9: warning: enum conversion when passing argument 1 of 'gen_rtx_MEM' is invalid in C++ [-Wc++-compat]
| /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/rtl.h:2259:12: note: expected 'enum machine_mode' but argument is of type 'int'
| /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/builtins.c: In function 'expand_builtin_update_setjmp_buf':
| /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/builtins.c:1116:8: warning: enum conversion in initialization is invalid in C++ [-Wc++-compat]
| gcc -c -isystem/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/sysroots/x86_64-linux/usr/include -O2 -pipe -DIN_GCC -DCROSS_DIRECTORY_STRUCTURE -W -Wall -Wwrite-strings -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes -Wmissing-format-attribute -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -Wold-style-definition -Wc++-compat -DHAVE_CONFIG_H -I. -I. -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/. -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/../include -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/../libcpp/include -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/../libdecnumber -I/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/../libdecnumber/dpd -I../libdecnumber /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/cfganal.c -o cfganal.o
| /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/calls.c: In function 'initialize_argument_information':
| /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/calls.c:1204:9: error: 'STACK_CHECK_MAX_VAR_SIZE' undeclared (first use in this function)
| /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work-shared/gcc-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/gcc/calls.c:1204:9: note: each undeclared identifier is reported only once for each function it appears in
| make[1]: *** [calls.o] Error 1
| make[1]: *** Waiting for unfinished jobs....
| rm gfdl.pod cpp.pod gcov.pod fsf-funding.pod gcc.pod
| make[1]: Leaving directory `/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work/aarch64-oe-linux/gcc-cross-initial-linaro-4.7-r6/gcc-linaro-4.7+bzr115029/build.x86_64-linux.aarch64-oe-linux/gcc'
| make: *** [all-gcc] Error 2
| ERROR: oe_runmake failed
| ERROR: Function failed: do_compile (see /mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/build/tmp-eglibc/work/aarch64-oe-linux/gcc-cross-initial-linaro-4.7-r6/temp/log.do_compile.14734 for further information)
NOTE: recipe gcc-cross-initial-linaro-4.7-r6: task do_compile: Failed
ERROR: Task 361 (/mnt/ci_build/workspace/openembedded-armv8-minimal/builddir/meta-linaro/recipes-devtools/gcc/gcc-cross-initial_linaro-4.7.bb, do_compile) failed with exit code '1'

Other folks on the internet complain about this in OE as well:
http://lists.linuxtogo.org/pipermail/openembedded-core/2011-December/014728.html
but no solution there; they point at race conditions.

Help from toolchain folks definitely welcome.

Cheers,

[Michael Hope (michaelh1)]

Note that the mailing list post suggests that the race is caused by an OE specific patch.

[Riku Voipio (riku-voipio)]

This seems to have suddenly resurfaced. I Was told on irc:

> yes, patching gcc is rather fraught because the source is shared between gcc-cross and gcc-cross-initial
> if you make sure that the SRC_URIs are identical (i.e. have the same patches) for all the recipes that use the same area in work-shared then you should be ok. if they are different then any amount of badness may ensue.

However, we have same SRC_URI for all (set in gcc-linaro-4.8.inc).