gcc-linaro uses an unreasonable amount of memory to compile qemu on armel

Reported by Steve Langasek on 2011-02-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Linaro GCC
Medium
Richard Sandiford
gcc-4.5 (Debian)
Fix Released
Unknown
gcc-4.5 (Ubuntu)
Undecided
Unassigned

Bug Description

I've been trying to sort out why qemu-linaro was failing to build on armel; the build logs have been showing unreasonable memory consumption - e.g., an OOM on a buildd with 30GB of swap. The same build did not run out of memory with gcc-4.4 on armel (I can reliably build the same code with ulimit -S -v $((1024*1024)) using gcc-4.4); it does not run out of memory when using the *same* gcc-4.5 as an x86->armel cross-compiler (requires a ulimit of 3250MiB in my tests); and it does not run out of memory on armel with gcc-4.5 if I pass -fno-var-tracking (completes with a ulimit of 320MiB).

But if I build with -O2 -g natively on armel and leave var-tracking on, it OOMs even with a ulimit of 5GiB.

This is more than a 16-fold increase in memory consumption from var tracking. Furthermore, the memory usage seems to show a dramatic spike right before the failure. Here's the memory usage immediately before the OOM:

# free
             total used free shared buffers cached
Mem: 496996 456488 40508 0 172 7616
-/+ buffers/cache: 448700 48296
Swap: 6032396 1411232 4621164
# ps u 2642
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2642 15.3 81.9 1615736 407472 ttyO2 T 22:40 11:00 /usr/lib/gcc/ar
#

This is 23 minutes into the build. But with a ulimit of 5GiB, within seconds I get the inevitable:

virtual memory exhausted: Cannot allocate memory

This is sudden enough that I'm convinced there's some kind of bug here; I can't imagine why gcc should need to allocate > 3GB of memory at a go.

I'll post a copy shortly of the preprocessed source used to reproduce this, plus the commandline that triggers it. I'll also see what I can root out with valgrind and/or strace; though at 24minutes an iteration it'll take a while yet.

Steve Langasek (vorlon) wrote :

Nonreduced preprocessed source attached. Build with this command (natively on arm) to reproduce:

/usr/lib/gcc/arm-linux-gnueabi/4.5.2/cc1 -quiet gcc-oom.i -quiet -dumpbase translate.c -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -mthumb -auxbase-strip translate.o -g -O2 -Werror -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wendif-labels -Wwrite-strings -Wmissing-prototypes -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fno-strict-aliasing -fstack-protector -o translate.s

$ gcc --version
gcc (Ubuntu/Linaro 4.5.2-2ubuntu1) 4.5.2
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$

The OOM comes while cc1 is writing the output file out to disk; strace shows a large number of writes interspersed with calls to mmap2() requesting 1MiB allocations for every ~680-748KiB output to the file. By the time of the OOM (with ulimit now raised to 6GiB), the output file has grown to 174MiB here; but that only accounts for something like 260MiB, nothing close to the ulimit. So I'm still not sure what's going on here.

Steve Langasek (vorlon) wrote :

Oh, and of course arm is 32-bit, so raising the ulimit above 4GiB will have no effect whatsoever. Heh. So perhaps this is why it works ok in the cross-compiler, since I'm cross-compiling from amd64.

Still, the cross-build required only 3250MiB, and snapshots of cc1 usage natively never show it exceeding 1.7GiB. Is there some in-kernel limit on the maximum per-process memory that we're hitting?

Is this behavior - allocating another 1MiB for every 700KiB written to disk - expected from gcc, or is this a buggy use of a buffer?

Peter Maydell (pmaydell) wrote :

The specific translate.c that is at issue here is target-sparc, used to build eg sparc64-linux-user/translate.o

On Sun, Feb 20, 2011 at 05:26:57PM -0000, Peter Maydell wrote:
> The specific translate.c that is at issue here is target-sparc, used to
> build eg sparc64-linux-user/translate.o

Actually, I've seen it with multiple different translate.c; the one I
submitted preprocessed source for should be the x86 translate.c.

Peter Maydell (pmaydell) wrote :

There's been a report on the qemu mailing list of this problem on a ppc32 host as well; again, -fno-var-tracking allowed the compile to succeed.

Michael Hope (michaelh1) on 2011-02-28
Changed in gcc-linaro:
assignee: nobody → Richard Sandiford (rsandifo)
Michael Hope (michaelh1) wrote :

Richard will attempt to reproduce on trunk, backport the patch if any, else report upstream.

Michael Hope (michaelh1) wrote :

Marking as triaged as we have sufficient information to reproduce.

Changed in gcc-linaro:
status: New → Triaged
importance: Undecided → Medium
Richard Sandiford (rsandifo) wrote :

Two patches posted upstream:

    http://gcc.gnu.org/ml/gcc-patches/2011-03/msg00193.html
    http://gcc.gnu.org/ml/gcc-patches/2011-03/msg00255.html

The first has also been committed upstream. The second is pending review, so I sent a ping today. I'll submit a merge request once both changes have gone in.

The first patch only applies to 4.6. The second (and main) one applies to both 4.5 and 4.6.

Michael Hope (michaelh1) on 2011-03-15
Changed in gcc-linaro:
status: Triaged → In Progress
Matthias Klose (doko) on 2011-03-20
Changed in gcc-4.5 (Ubuntu):
status: New → Confirmed
Changed in gcc-4.5 (Debian):
status: Unknown → New
Changed in gcc-4.5 (Debian):
status: New → Confirmed
Michael Hope (michaelh1) on 2011-04-15
Changed in gcc-linaro:
status: In Progress → Fix Committed
Michael Hope (michaelh1) wrote :

Richard, should we backport this to gcc-linaro 4.6 as well?

Changed in gcc-linaro:
milestone: none → 4.5-2011.05-0

I am out of the office until 17/04/2011.

Note: This is an automated response to your message "[Bug 714921] Re:
gcc-linaro uses an unreasonable amount of memory to compile qemu on
armel" sent on 15/4/2011 4:50:47.

This is the only notification you will receive while this person is away.

Michael Hope <email address hidden> writes:
> Richard, should we backport this to gcc-linaro 4.6 as well?

I'm still hoping to get this in upstream 4.6. But if that doesn't
work out, then yeah, we should make it linaro local.

Richard

Michael Hope (michaelh1) on 2011-05-19
Changed in gcc-linaro:
status: Fix Committed → Fix Released
Matthias Klose (doko) wrote :

the patch is applied in the natty gcc-4.5 package

Changed in gcc-4.5 (Ubuntu):
status: Confirmed → Fix Released
Changed in gcc-4.5 (Debian):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.