[PR40735] gcc-4.4 fails to build upstart 0.6 on armel due to an internal compiler error

Bug #398403 reported by Oliver Grawert on 2009-07-12
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Linaro GCC
Fix Released
Medium
Unassigned
gcc
Confirmed
Medium
gcc-4.4 (Ubuntu)
High
Unassigned
libnih (Ubuntu)
Undecided
Unassigned
upstart (Ubuntu)
Undecided
Unassigned

Bug Description

Binary package hint: gcc-4.4

in
https://edge.launchpad.net/ubuntu/+source/upstart/0.6.0-2/+build/1112005/+files/buildlog_ubuntu-karmic-armel.upstart_0.6.0-2_FAILEDTOBUILD.txt.gz
we see that the buidd appears to time out (in reality its hiding the actual error).
building that package on a local armel machine spits out the following:

/bin/bash ../libtool --tag=CC --mode=link gcc -std=gnu99 -Wall -g -fstack-protector -fPIE -Os -static -Wl,-z,relro -Wl,-z,now -pie -Wl,-O1 -o test_file test_file.o libnih.la
libtool: link: gcc -std=gnu99 -Wall -g -fstack-protector -fPIE -Os -Wl,-z -Wl,relro -Wl,-z -Wl,now -pie -Wl,-O1 -o test_file test_file.o ./.libs/libnih.a -lrt
gcc -std=gnu99 -DHAVE_CONFIG_H -DLOCALEDIR="\"/usr/share/locale\"" -I.. -I.. -I../intl -Wall -g -fstack-protector -fPIE -Os -MT test_watch.o -MD -MP -MF .deps/test_watch.Tpo -c -o test_watch.o `test -f 'tests/test_watch.c' || echo './'`tests/test_watch.c
tests/test_watch.c: In function ‘test_new’:
tests/test_watch.c:590: internal compiler error: in df_ref_record, at df-scan.c:2845
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.4/README.Bugs> for instructions.
make[3]: *** [test_watch.o] Error 1
make[3]: Leaving directory `/home/ogra/devel/upstart-0.6.0/nih'
make[2]: *** [check-am] Error 2
make[2]: Leaving directory `/home/ogra/devel/upstart-0.6.0/nih'
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory `/home/ogra/devel/upstart-0.6.0'

Oliver Grawert (ogra) wrote :

different place of the error in a different system with a local build, it seems to be closely related to the hardware gcc is running on ...

gcc -std=gnu99 -DHAVE_CONFIG_H -DLOCALEDIR="\"/usr/share/locale\"" -I.. -I.. -iquote. -iquote. -I../intl -I/usr/include/dbus-1.0 -I/usr/lib/dbus-1.0/include -Wall -g -fstack-protector -fPIE -Os -MT test_node.o -MD -MP -MF .deps/test_node.Tpo -c -o test_node.o `test -f 'tests/test_node.c' || echo './'`tests/test_node.c
gcc: Internal error: Killed (program cc1)
Please submit a full bug report.
See <file:///usr/share/doc/gcc-4.4/README.Bugs> for instructions.
make[4]: *** [test_node.o] Error 1
make[4]: Leaving directory `/home/ogra/upstart-0.6.0/nih-dbus-tool'
make[3]: *** [check-am] Error 2
make[3]: Leaving directory `/home/ogra/upstart-0.6.0/nih-dbus-tool'
make[2]: *** [check] Error 2
make[2]: Leaving directory `/home/ogra/upstart-0.6.0/nih-dbus-tool'
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory `/home/ogra/upstart-0.6.0'

Changed in gcc-4.4 (Ubuntu):
importance: Undecided → High
tags: added: armel
Oliver Grawert (ogra) wrote :

downgrading my local buildsystem to gcc-4.3 i see similar issues

Perhaps it was forced to swap or ran out of virtual memory. Sometimes
you will find some strange situation in some piece of code where gcc
seems to decide it needs a massive amount of memory and a very long time
to compile something optimized. For example, there was a time when
compiling the pci code in X.org for powerpc would require more than 1
gig of ram, but only to compile one very specific file, and only as long
as optimization was used. Maybe we are seeing something similar here?

Oliver Grawert wrote:
> Public bug reported:
>
> Binary package hint: gcc-4.4
>
> in
> https://edge.launchpad.net/ubuntu/+source/upstart/0.6.0-2/+build/1112005/+files/buildlog_ubuntu-karmic-armel.upstart_0.6.0-2_FAILEDTOBUILD.txt.gz
> we see that the buidd appears to time out (in reality its hiding the actual error).
> building that package on a local armel machine spits out the following:
>
>
> /bin/bash ../libtool --tag=CC --mode=link gcc -std=gnu99 -Wall -g -fstack-protector -fPIE -Os -static -Wl,-z,relro -Wl,-z,now -pie -Wl,-O1 -o test_file test_file.o libnih.la
> libtool: link: gcc -std=gnu99 -Wall -g -fstack-protector -fPIE -Os -Wl,-z -Wl,relro -Wl,-z -Wl,now -pie -Wl,-O1 -o test_file test_file.o ./.libs/libnih.a -lrt
> gcc -std=gnu99 -DHAVE_CONFIG_H -DLOCALEDIR="\"/usr/share/locale\"" -I.. -I.. -I../intl -Wall -g -fstack-protector -fPIE -Os -MT test_watch.o -MD -MP -MF .deps/test_watch.Tpo -c -o test_watch.o `test -f 'tests/test_watch.c' || echo './'`tests/test_watch.c
> tests/test_watch.c: In function ‘test_new’:
> tests/test_watch.c:590: internal compiler error: in df_ref_record, at df-scan.c:2845
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <file:///usr/share/doc/gcc-4.4/README.Bugs> for instructions.
> make[3]: *** [test_watch.o] Error 1
> make[3]: Leaving directory `/home/ogra/devel/upstart-0.6.0/nih'
> make[2]: *** [check-am] Error 2
> make[2]: Leaving directory `/home/ogra/devel/upstart-0.6.0/nih'
> make[1]: *** [check-recursive] Error 1
> make[1]: Leaving directory `/home/ogra/devel/upstart-0.6.0'
>
> ** Affects: gcc-4.4 (Ubuntu)
> Importance: High
> Status: New
>
>
> ** Tags: armel
>
> ** Changed in: gcc-4.4 (Ubuntu)
> Importance: Undecided => High
>
> ** Tags added: armel
>

David, please try rebuilding locally and see if you run out of memory; I doubt it's the case though.

Any of you reproducing the bug with latest compiler, please collect the generated file which triggers the crash as explained in the gcc debug instructions.

Oliver Grawert (ogra) wrote :

i doubt that its a memory issue and i see it on all SoCs here
i will try building gcc-snapshot and see if it occus with that too ...
for both SoCs i have here, i only have binary kernels, it might be cause by some ulimit issue ...

there should be enough swap here and the system is running (and swapping) from a SATA disk

ogra@dove:~/devel/upstart-0.6.0$ free
             total used free shared buffers cached
Mem: 514740 435284 79456 0 55104 306752
-/+ buffers/cache: 73428 441312
Swap: 3903784 168 3903616

Loïc Minier (lool) wrote :

So it's actually gcc using a load of memory indeed; it seems that it doesn't like the 16k lines of code in test_node.c and there are larger files to build after that.

Loïc Minier (lool) wrote :

19:06 < lool> cjwatson: I built with -O0 -fno-common -fno-foo ... disabling all
              enabled opts until the output would only mention one enabled opt
              which is the reverse of another one; the build still takes
              hundreds of megs of RAM and I stop it at around 80 % RAM after 3
              minutes
19:07 < lool> I guess I should have tried with an unpatched gcc though
...
19:08 < lool> Oh it passed
19:08 < lool> after 4 minutes or so
19:09 < lool> but using a big load of RAM
...
19:12 < lool> doko: I don't quite know how to attack the upstart build slowness
              issue
19:12 < lool> Basically, we're seeing underperformance, but it's hard to
              qualify how fast gcc should run, or how much memory it's allowed
              to use
19:12 < lool> doko: Do you think some patches we're using could be destroying
              performance?
19:12 < lool> e.g. that 64 bytes align patch
19:13 < lool> That's a binutils patch, but I can see how it could lead to much
              larger RAM usage
19:13 < doko> lool: where is the time used, in cc1, or as?
19:13 < lool> doko: cc1
19:14 < doko> well, then it's not the align patch, and we are pretty much using
              upstream, no fancy patches on our own
19:15 < lool> doko: Can I just report it upsteam opening a new bug saying that
              it uses a load of RAM/CPU to build a file?
19:15 < lool> ie Is that an acceptable bug?
19:16 < doko> lool: sure, better check it with 4.3 (or older) as well to see if
              it's a regression or not.
...
19:23 < lool> doko: It's about the same performance with gcc-4.3

This is so very much like my strange experience last year with just a
very specific file in X.org that would similarly balloon gcc memory
usage cross-compiling for powerpc...the solution chosen at the time was
to use 2G or larger build machines ;).

Loïc Minier wrote:
> 19:06 < lool> cjwatson: I built with -O0 -fno-common -fno-foo ... disabling all
> enabled opts until the output would only mention one enabled opt
> which is the reverse of another one; the build still takes
> hundreds of megs of RAM and I stop it at around 80 % RAM after 3
> minutes
> 19:07 < lool> I guess I should have tried with an unpatched gcc though
> ...
> 19:08 < lool> Oh it passed
> 19:08 < lool> after 4 minutes or so
> 19:09 < lool> but using a big load of RAM
> ...
> 19:12 < lool> doko: I don't quite know how to attack the upstart build slowness
> issue
> 19:12 < lool> Basically, we're seeing underperformance, but it's hard to
> qualify how fast gcc should run, or how much memory it's allowed
> to use
> 19:12 < lool> doko: Do you think some patches we're using could be destroying
> performance?
> 19:12 < lool> e.g. that 64 bytes align patch
> 19:13 < lool> That's a binutils patch, but I can see how it could lead to much
> larger RAM usage
> 19:13 < doko> lool: where is the time used, in cc1, or as?
> 19:13 < lool> doko: cc1
> 19:14 < doko> well, then it's not the align patch, and we are pretty much using
> upstream, no fancy patches on our own
> 19:15 < lool> doko: Can I just report it upsteam opening a new bug saying that
> it uses a load of RAM/CPU to build a file?
> 19:15 < lool> ie Is that an acceptable bug?
> 19:16 < doko> lool: sure, better check it with 4.3 (or older) as well to see if
> it's a regression or not.
> ...
> 19:23 < lool> doko: It's about the same performance with gcc-4.3
>

not building the file with -fPIE limits the memory usage to a maximum of 600MB, approx. 10min build time.

[forwarded from https://launchpad.net/bugs/398403]

Building the attached file with gcc from the 4.4 branch with -g -fstack-protector -fPIE -Os, the build fails (killed by oom), last info seen is memory usage of about 1500mb. Building without -fPIE memory usage is limited around 700MB.

Same behaviour for 4.3. With gcc-4.2 peak memory usage is with/without -fPIE below 400MB. Not checked on trunk, currently fails to build.

Created attachment 18188
preprocessed source

Matthias Klose (doko) wrote :

regression in gcc-4.3/gcc-4.4.

Matthias Klose (doko) wrote :

still unsure why these functions in the test cases cannot be split up into smaller functions which then are called sequentially.

Changed in gcc:
status: Unknown → New
Matthias Klose (doko) on 2009-07-13
Changed in gcc-4.4 (Ubuntu):
status: New → Confirmed
Loïc Minier (lool) wrote :

18:24 < lool> Keybuk: Couldn't we split at least the
              test_path_valid/test_new/test_start_tag etc. into separate files?
18:24 < Keybuk> lool: no, the test suite doesn't work that way
18:24 < lool> test_proxy_functions is the huge function; 10k lines
18:25 < lool> Keybuk: What do you suggest?
18:25 < Keybuk> lool: fix gcc

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package upstart - 0.6.0-4

---------------
upstart (0.6.0-4) karmic; urgency=low

  * Don't build the testsuite with -fPIE on armel; LP: #398403.

 -- Loic Minier <email address hidden> Mon, 13 Jul 2009 22:12:34 +0200

Changed in upstart (Ubuntu):
status: New → Fix Released

I cannot compile the attached testcase.

gcc-4.2 -S -o /dev/null -g -fstack-protector -fPIE -Os test_node.i -std=c99
In file included from ../nih/test_alloc.h:32,
                 from ../nih/test.h:36,
                 from tests/test_node.c:23:
../nih/list.h:110: warning: declaration does not declare anything
In file included from tests/test_node.c:40:
./parse.h:71: warning: declaration does not declare anything
tests/test_node.c: In function ‘test_start_tag’:
tests/test_node.c:211: error: ‘ParseStack’ has no member named ‘node’
tests/test_node.c:251: error: ‘ParseStack’ has no member named ‘node’
tests/test_node.c:297: error: ‘ParseStack’ has no member named ‘data’
tests/test_node.c:297: error: ‘ParseStack’ has no member named ‘data’
tests/test_node.c:365: error: ‘ParseStack’ has no member named ‘node’
tests/test_node.c:422: error: ‘ParseStack’ has no member named ‘data’
tests/test_node.c:422: error: ‘ParseStack’ has no member named ‘data’
tests/test_node.c: In function ‘test_object_functions’:
tests/test_node.c:1256: error: ‘NihListEntry’ has no member named ‘str’
tests/test_node.c:1256: error: ‘NihListEntry’ has no member named ‘str’
...

my bad, should be -std=gnu99 instead: gcc-4.2 -S -o /dev/null -g -fstack-protector -fPIE -Os test_node.i -std=gnu99

This is likely caused by the DF merge. There are numerous bugs about this
already and nothing really can be done here.

Btw, my numbers are

rguenther@murzim:/tmp> ~/bin/maxmem2.sh gcc-4.4 -S -o /dev/null -g -fstack-protector -fPIE -Os test_node.i -std=gnu99
total: 744108 kB
rguenther@murzim:/tmp> ~/bin/maxmem2.sh gcc-4.3 -S -o /dev/null -g -fstack-protector -fPIE -Os test_node.i -std=gnu99
total: 719836 kB
rguenther@murzim:/tmp> ~/bin/maxmem2.sh gcc-4.2 -S -o /dev/null -g -fstack-protector -fPIE -Os test_node.i -std=gnu99
total: 459757 kB

Thus not 1.5GB but 750MB vs 450MB.

rguenther@murzim:/tmp> ~/bin/maxmem2.sh gcc-4.4 -S -o /dev/null -fPIE -Os test_node.i -std=gnu99
total: 743380 kB
rguenther@murzim:/tmp> ~/bin/maxmem2.sh gcc-4.4 -S -o /dev/null -Os test_node.i -std=gnu99
total: 630756 kB

the -fPIE effect itself is even less recognizable.

In , Mikpe (mikpe) wrote :

Seems heavily target-dependent. With 4.3.4 I see peak usage of 640M for i686, just under 1.2G for powerpc, and 1.6G for arm.

Hmm, maybe scheduling is causing it. Does -fno-schedule-insns -fno-schedule-insns2 cause the memory usage to go down?

In , Mikpe (mikpe) wrote :

(In reply to comment #6)
> Hmm, maybe scheduling is causing it. Does -fno-schedule-insns
> -fno-schedule-insns2 cause the memory usage to go down?

Nope, memory usage patterns stayed the same.

Does seem to be a real issue, somewhere.
When trunk builds again, can you please give it a try too?

DF time on this testcase is already big, and the testcase has lots of function
calls which would explain the difference between targets (DF needs to track all
call-used/clobbered regs).

Changed in gcc:
status: New → Confirmed
In , Mikpe (mikpe) wrote :

More memory usage numbers on this test case:

With 4.4.1-RC-20090715: i686 peaks at 616M, powerpc at 799M, and arm at 1211M.

With 4.5.0-20090709: i686 peaks at 530M, powerpc at 707M, and arm at 933M.

GCC 4.3.4 is being released, adjusting target milestone.

Matthias Klose (doko) on 2010-01-04
summary: - gcc-4.4 fails to build upstart 0.6 on armel due to an internal compiler
- error
+ [PR40735] gcc-4.4 fails to build upstart 0.6 on armel due to an internal
+ compiler error

GCC 4.3.5 is being released, adjusting target milestone.

Matthias Klose (doko) wrote :

libnih build fails in maverick on armel

Changed in libnih (Ubuntu):
status: New → Confirmed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libnih - 1.0.2-1ubuntu1

---------------
libnih (1.0.2-1ubuntu1) maverick; urgency=low

  * Rebuild with libc6-dev (>= 2.12~), after checking that
    __abort_msg is available with the same signature in eglibc 2.12.
  * Don't build the testsuite with -fPIE on armel; LP: #398403.
 -- Matthias Klose <email address hidden> Sun, 30 May 2010 02:54:56 +0200

Changed in libnih (Ubuntu):
status: Confirmed → Fix Released
Matthias Klose (doko) on 2010-06-22
tags: added: toolchain
Changed in gcc-linaro:
importance: Undecided → Medium
Loïc Minier (lool) wrote :

I uploaded upstart back without -fPIE / -pie, just to see how that goes with Linaro toolchain; otherwise, I think we should fix that.

On 23.07.2010 17:07, Loïc Minier wrote:
> I uploaded upstart back without -fPIE / -pie, just to see how that goes
> with Linaro toolchain; otherwise, I think we should fix that.

it's enough to build the *testcases* without pie to pass the testsuite.

Loïc Minier (lool) wrote :

Yes, but someone changed the packaging to remove it everywhere instead of just for the testsuite, and also I'm curious of the gcc issue -- not the testsuite one, I think this is fine.

Loïc Minier (lool) wrote :

So the new upstart upload I did fails to build in the testsuite as on the other architectures, but most tests DID build with -fPIE, some 3 tests or so remained to be built. test_watch.c isn't in upstart anymore.

I tried removing -fPIE/-pie in libnih, and it failed to build with the maverick Linaro-based toolchain; I can't access the build log to check how it failed though.

Loïc Minier (lool) wrote :

Ah just as said that, I could access the log:
gcc -std=gnu99 -DHAVE_CONFIG_H -DLOCALEDIR="\"/usr/share/locale\"" -I.. -I.. -iquote. -iquote. -I../intl -I/usr/include/dbus-1.0 -I/usr/lib/dbus-1.0/include -Wall -g -fstack-protector -fPIE -Os -c -o demarshal_code.o `test -f 'tests/demarshal_code.c' || echo './'`tests/demarshal_code.c
  CCLD test_demarshal
  CC test_node.o
gcc -std=gnu99 -DHAVE_CONFIG_H -DLOCALEDIR="\"/usr/share/locale\"" -I.. -I.. -iquote. -iquote. -I../intl -I/usr/include/dbus-1.0 -I/usr/lib/dbus-1.0/include -Wall -g -fstack-protector -fPIE -Os -c -o test_node.o `test -f 'tests/test_node.c' || echo './'`tests/test_node.c

Session terminated, killing shell...make: *** [build] Terminatedmake[1]: *** wait: No child processes
. Stop.
make[1]: *** Waiting for unfinished jobs....
[...]
Build killed with signal 15 after 150 minutes of inactivity

It might be an issue with the cheer size of the file, but it didn't segfault at least:
  CC test_watch.o
gcc -std=gnu99 -DHAVE_CONFIG_H -DLOCALEDIR="\"/usr/share/locale\"" -I.. -I.. -I../intl -Wall -g -fstack-protector -fPIE -Os -c -o test_watch.o `test -f 'tests/test_watch.c' || echo './'`tests/test_watch.c
  CCLD test_watch

I've just built upstart_0.6.6-1ubuntu1.dsc on an IGEPv2 board.

The build appears to succeed and run as far as testing, but the tests fail:

....
FAIL: test_job_class
....
FAIL: test_job
....
FAIL: test_control
===============================================
3 of 13 tests failed
Please report to <email address hidden>
===============================================
make[4]: *** [check-TESTS] Error 1
make[4]: Leaving directory `/home/ams/lp398403/upstart-0.6.6/init'
make[3]: *** [check-am] Error 2
make[3]: Leaving directory `/home/ams/lp398403/upstart-0.6.6/init'
make[2]: *** [check] Error 2
make[2]: Leaving directory `/home/ams/lp398403/upstart-0.6.6/init'
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory `/home/ams/lp398403/upstart-0.6.6'
dh_auto_test: make -j1 check returned exit code 2
make: *** [build] Error 29
dpkg-buildpackage: error: debian/rules build gave error exit status 2

Michael Hope (michaelh1) wrote :

I added -fpie back in to a local build of libnih 1.0.2-1ubuntu1 and it built and tested correctly. Compiling test_node.c takes about 1 GB of RAM.

This is with maverick and gcc-4.4-4.4.4-8ubuntu1.

Loïc Minier (lool) wrote :

So apparently fixed in some newer gcc-4.4; thanks for testing

i've just uploaded libnih without the armel specific behavior of building the testsuite with -fPIE.

Changed in gcc-4.4 (Ubuntu):
status: Confirmed → Fix Released
Changed in gcc-linaro:
status: New → Fix Released

4.3 branch is being closed, moving to 4.4.7 target.

Changed in gcc:
importance: Unknown → Medium

(In reply to comment #4)
> This is likely caused by the DF merge. There are numerous bugs about this
> already and nothing really can be done here.
>
> Btw, my numbers are
>
> rguenther@murzim:/tmp> ~/bin/maxmem2.sh gcc-4.4 -S -o /dev/null -g
> -fstack-protector -fPIE -Os test_node.i -std=gnu99
> total: 744108 kB
> rguenther@murzim:/tmp> ~/bin/maxmem2.sh gcc-4.3 -S -o /dev/null -g
> -fstack-protector -fPIE -Os test_node.i -std=gnu99
> total: 719836 kB
> rguenther@murzim:/tmp> ~/bin/maxmem2.sh gcc-4.2 -S -o /dev/null -g
> -fstack-protector -fPIE -Os test_node.i -std=gnu99
> total: 459757 kB

> ~/bin/maxmem2.sh gcc-4.6 -S -o /dev/null t.i -v -std=gnu99 -Os -fPIE -fstack-protector -g
total: 772239 kB
> ~/bin/maxmem2.sh gcc-4.7 -S -o /dev/null t.i -v -std=gnu99 -Os -fPIE -fstack-protector -g --param ggc-min-expand=100 --param ggc-min-heapsize=131072
total: 794883 kB

we're continuing to increase, thus re-confirmed.

Blocks the memory-hog meta-bug which is marked as regression.

obtain head

(In reply to comment #4)
> ~/bin/maxmem2.sh gcc-4.4 -S -o /dev/null -g
> -fstack-protector -fPIE -Os test_node.i -std=gnu99
> total: 744108 kB

richi, can you share this maxmem2 script?

(In reply to comment #17)
>
> richi, can you share this maxmem2 script?

It's available on the wiki: http://gcc.gnu.org/wiki/PerformanceTesting

With r190665, compile time is mostly spent in:
 loop doloop : 130.15 ( 8%) usr
 variable tracking : 990.72 (60%) usr
 var-tracking dataflow : 76.34 ( 5%) usr

So this isn't a DF issue in GCC 4.8 at least, but yet another var-tracking issue (but probably not related to bug 53958, because var-tracking dataflow time is almost reasonable here).

I'll have a look at the doloop issue, but the var-tracking issue is for Alex.

With release checking, the loop-doloop time is almost zero (so something is slow in doloop checking). The var-tracking slowness does not disappear, and dominates the compile time even more strongly:

 variable tracking : 530.57 (84%) usr

(In reply to comment #14)
> > ~/bin/maxmem2.sh gcc-4.7 -S -o /dev/null t.i -v -std=gnu99 -Os -fPIE -fstack-protector -g --param ggc-min-expand=100 --param ggc-min-heapsize=131072
> total: 794883 kB
>
> we're continuing to increase, thus re-confirmed.

At r190665, with the same -min-expand=100 --param ggc-min-heapsize=131072 :

tests/test_node.c:6115:1: note: variable tracking size limit exceeded with -fvar-tracking-assignments, retrying without
$ total: 1443401 kB

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.