MongoDB triggers OOM killer on autobuilders

[Fathi Boudra]

https://ci.linaro.org/jenkins/job/openembedded-armv8-rootfs/gcc_version=4.8,label=build,rootfs=leg-java/650/consoleFull
https://ci.linaro.org/jenkins/job/openembedded-armv8-rootfs/gcc_version=4.8,label=build,rootfs=lamp/654/consoleFull

TCWG asked for a few things to track down the issue:
* could you retry the g++ command which failed with -save-temps, and send me the hashcmd.i which results?
looking again at the log, I think it is mr.cpp which failed, rather than hashcmd.cpp, so try the command line for that file
with -save-temps

[Fathi Boudra (fboudra)]

Charles, I re-triggered the build to include -save-temps and get the information you wanted but it didn't failed on the same file.

| aarch64-oe-linux-g++ --sysroot=/srv/oe-fabo/jenkins-setup/build/tmp-eglibc/sysroots/genericarmv8 -o build/linux2/disable-scripting/ld_aarch64-oe-linux-g++/ssl/use-system-boost/use-system-pcre/use-system-tcmalloc/mongo/db/repl/replication_server_status.o -c -Wnon-virtual-dtor -Woverloaded-virtual -fPIC -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -Werror -pipe -O3 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -fno-builtin-memcmp -D_SCONS -DMONGO_EXPOSE_MACROS -DSUPPORT_UTF8 -DMONGO_OPTIMIZED_BUILD -D_FILE_OFFSET_BITS=64 -DMONGO_SSL -DMONGO_HAVE___THREAD -DBOOST_THREAD_VERSION=2 -DMONGO_HAVE_HEADER_UNISTD_H -DMONGO_HAVE_EXECINFO_BACKTRACE -Ibuild/linux2/disable-scripting/ld_aarch64-oe-linux-g++/ssl/use-system-boost/use-system-pcre/use-system-tcmalloc/third_party/snappy -Isrc/third_party/snappy -Ibuild/linux2/disable-scripting/ld_aarch64-oe-linux-g++/ssl/use-system-boost/use-system-pcre/use-system-tcmalloc/third_party/libstemmer_c/include -Isrc/third_party/libstemmer_c/include -Ibuild/linux2/disable-scripting/ld_aarch64-oe-linux-g++/ssl/use-system-boost/use-system-pcre/use-system-tcmalloc/third_party/s2 -Isrc/third_party/s2 -Ibuild/linux2/disable-scripting/ld_aarch64-oe-linux-g++/ssl/use-system-boost/use-system-pcre/use-system-tcmalloc -Isrc -Ibuild/linux2/disable-scripting/ld_aarch64-oe-linux-g++/ssl/use-system-boost/use-system-pcre/use-system-tcmalloc/mongo -Isrc/mongo -Ibuild/linux2/disable-scripting/ld_aarch64-oe-linux-g++/ssl/use-system-boost/use-system-pcre/use-system-tcmalloc/third_party/s2 -Isrc/third_party/s2 -Ibuild/linux2/disable-scripting/ld_aarch64-oe-linux-g++/ssl/use-system-boost/use-system-pcre/use-system-tcmalloc/third_party/snappy -Isrc/third_party/snappy -Ibuild/linux2/disable-scripting/ld_aarch64-oe-linux-g++/ssl/use-system-boost/use-system-pcre/use-system-tcmalloc/third_party/yaml-cpp-0.5.1/include -Isrc/third_party/yaml-cpp-0.5.1/include src/mongo/db/repl/replication_server_status.cpp
| {standard input}: Assembler messages:
| {standard input}:62201: Warning: partial line at end of file ignored
| {standard input}:62173: Error: invalid operands (*UND* and .gcc_except_table sections) for `-'
| {standard input}:62176: Error: invalid operands (*UND* and .gcc_except_table sections) for `-'
| aarch64-oe-linux-g++: internal compiler error: Killed (program cc1plus)
| Please submit a full bug report,
| with preprocessed source if appropriate.
| See <http://gcc.gnu.org/bugs.html> for instructions.
| scons: *** [build/linux2/disable-scripting/ld_aarch64-oe-linux-g++/ssl/use-system-boost/use-system-pcre/use-system-tcmalloc/mongo/db/instance.o] Error 4
| scons: building terminated because of errors.

[Will Newton (will-newton)]

Do you have access to dmesg on this machine? Is the OOM killer killing the process perhaps?

[Fathi Boudra (fboudra)]

Good catch!

Out of memory in UB 20563: OOM killed process 1576 (cc1plus) score 0 vm:592448kB, rss:547436kB, swap:0kB

[Charles Baylis (cbaylis)]

I think the .i file from any compilation which has failed will still be useful to see which part of the compiler is responsible for the high memory usage.

If you attach replication_server_status.i, I'll take a look.

[Fathi Boudra (fboudra)]

wontfix. switching to a dedicated server with more RAM.