gcc-linaro uses an unreasonable amount of memory to compile qemu on armel

[Steve Langasek]

I've been trying to sort out why qemu-linaro was failing to build on armel; the build logs have been showing unreasonable memory consumption - e.g., an OOM on a buildd with 30GB of swap. The same build did not run out of memory with gcc-4.4 on armel (I can reliably build the same code with ulimit -S -v $((1024*1024)) using gcc-4.4); it does not run out of memory when using the *same* gcc-4.5 as an x86->armel cross-compiler (requires a ulimit of 3250MiB in my tests); and it does not run out of memory on armel with gcc-4.5 if I pass -fno-var-tracking (completes with a ulimit of 320MiB).

But if I build with -O2 -g natively on armel and leave var-tracking on, it OOMs even with a ulimit of 5GiB.

This is more than a 16-fold increase in memory consumption from var tracking. Furthermore, the memory usage seems to show a dramatic spike right before the failure. Here's the memory usage immediately before the OOM:

# free
             total used free shared buffers cached
Mem: 496996 456488 40508 0 172 7616
-/+ buffers/cache: 448700 48296
Swap: 6032396 1411232 4621164
# ps u 2642
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2642 15.3 81.9 1615736 407472 ttyO2 T 22:40 11:00 /usr/lib/gcc/ar
#

This is 23 minutes into the build. But with a ulimit of 5GiB, within seconds I get the inevitable:

virtual memory exhausted: Cannot allocate memory

This is sudden enough that I'm convinced there's some kind of bug here; I can't imagine why gcc should need to allocate > 3GB of memory at a go.

I'll post a copy shortly of the preprocessed source used to reproduce this, plus the commandline that triggers it. I'll also see what I can root out with valgrind and/or strace; though at 24minutes an iteration it'll take a while yet.

[Steve Langasek (vorlon)]

Nonreduced preprocessed source attached. Build with this command (natively on arm) to reproduce:

/usr/lib/gcc/arm-linux-gnueabi/4.5.2/cc1 -quiet gcc-oom.i -quiet -dumpbase translate.c -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -mthumb -auxbase-strip translate.o -g -O2 -Werror -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wendif-labels -Wwrite-strings -Wmissing-prototypes -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fno-strict-aliasing -fstack-protector -o translate.s

$ gcc --version
gcc (Ubuntu/Linaro 4.5.2-2ubuntu1) 4.5.2
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$

The OOM comes while cc1 is writing the output file out to disk; strace shows a large number of writes interspersed with calls to mmap2() requesting 1MiB allocations for every ~680-748KiB output to the file. By the time of the OOM (with ulimit now raised to 6GiB), the output file has grown to 174MiB here; but that only accounts for something like 260MiB, nothing close to the ulimit. So I'm still not sure what's going on here.

[Steve Langasek (vorlon)]

Oh, and of course arm is 32-bit, so raising the ulimit above 4GiB will have no effect whatsoever. Heh. So perhaps this is why it works ok in the cross-compiler, since I'm cross-compiling from amd64.

Still, the cross-build required only 3250MiB, and snapshots of cc1 usage natively never show it exceeding 1.7GiB. Is there some in-kernel limit on the maximum per-process memory that we're hitting?

Is this behavior - allocating another 1MiB for every 700KiB written to disk - expected from gcc, or is this a buggy use of a buffer?

[Peter Maydell (pmaydell)]

The specific translate.c that is at issue here is target-sparc, used to build eg sparc64-linux-user/translate.o

[Steve Langasek (vorlon)]

On Sun, Feb 20, 2011 at 05:26:57PM -0000, Peter Maydell wrote:
> The specific translate.c that is at issue here is target-sparc, used to
> build eg sparc64-linux-user/translate.o

Actually, I've seen it with multiple different translate.c; the one I
submitted preprocessed source for should be the x86 translate.c.

[Peter Maydell (pmaydell)]

There's been a report on the qemu mailing list of this problem on a ppc32 host as well; again, -fno-var-tracking allowed the compile to succeed.

[Michael Hope (michaelh1)]

Richard will attempt to reproduce on trunk, backport the patch if any, else report upstream.

[Michael Hope (michaelh1)]

Marking as triaged as we have sufficient information to reproduce.

[Richard Sandiford (rsandifo)]

Two patches posted upstream:

    http://gcc.gnu.org/ml/gcc-patches/2011-03/msg00193.html
    http://gcc.gnu.org/ml/gcc-patches/2011-03/msg00255.html

The first has also been committed upstream. The second is pending review, so I sent a ping today. I'll submit a merge request once both changes have gone in.

The first patch only applies to 4.6. The second (and main) one applies to both 4.5 and 4.6.

[Michael Hope (michaelh1)]

Richard, should we backport this to gcc-linaro 4.6 as well?

[Ira Rosen (irar)]

I am out of the office until 17/04/2011.

Note: This is an automated response to your message "[Bug 714921] Re:
gcc-linaro uses an unreasonable amount of memory to compile qemu on
armel" sent on 15/4/2011 4:50:47.

This is the only notification you will receive while this person is away.

[Richard Sandiford (rsandifo)]

Michael Hope <email address hidden> writes:
> Richard, should we backport this to gcc-linaro 4.6 as well?

I'm still hoping to get this in upstream 4.6. But if that doesn't
work out, then yeah, we should make it linaro local.

Richard

[Matthias Klose (doko)]

the patch is applied in the natty gcc-4.5 package