postgres segfaults

[LoOoD]

Postgres will occasionall segfault every few weeks. I was able to get a core dump and get a backstrace. I'm running postgres-8.4 version 8.4.9-0ubuntu0.10.04 .

puppet:~$ gdb /usr/lib/postgresql/8.4/bin/postgres core
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/lib/postgresql/8.4/bin/postgres...Reading symbols from /usr/lib/debug/usr/lib/postgresql/8.4/bin/postgres...done.
done.
[New Thread 30165]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /usr/lib/libxml2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/libpam.so.0...Reading symbols from /usr/lib/debug/lib/libpam.so.0.82.2...done.
done.
Loaded symbols for /lib/libpam.so.0
Reading symbols from /lib/libssl.so.0.9.8...Reading symbols from /usr/lib/debug/lib/libssl.so.0.9.8...done.
done.
Loaded symbols for /lib/libssl.so.0.9.8
Reading symbols from /lib/libcrypto.so.0.9.8...Reading symbols from /usr/lib/debug/lib/libcrypto.so.0.9.8...done.
done.
Loaded symbols for /lib/libcrypto.so.0.9.8
Reading symbols from /usr/lib/libkrb5.so.3...Reading symbols from /usr/lib/debug/usr/lib/libkrb5.so.3.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /lib/libdl.so.2...Reading symbols from /usr/lib/debug/lib/libdl-2.11.1.so...done.
done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.so.6...Reading symbols from /usr/lib/debug/lib/libm-2.11.1.so...done.
done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /lib/libc.so.6...Reading symbols from /usr/lib/debug/lib/libc-2.11.1.so...done.
done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/libcrypt.so.1...Reading symbols from /usr/lib/debug/lib/libcrypt-2.11.1.so...done.
done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /lib/libresolv.so.2...Reading symbols from /usr/lib/debug/lib/libresolv-2.11.1.so...done.
done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib/ld-2.11.1.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/libsasl2.so.2...Reading symbols from /usr/lib/debug/usr/lib/libsasl2.so.2.0.23...done.
done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libgnutls.so.26...Reading symbols from /usr/lib/debug/usr/lib/libgnutls.so.26.14.12...done.
done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /usr/lib/libtasn1.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /lib/libgcrypt.so.11...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcrypt.so.11
Reading symbols from /lib/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libgpg-error.so.0
Reading symbols from /lib/libnss_compat.so.2...Reading symbols from /usr/lib/debug/lib/libnss_compat-2.11.1.so...done.
done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnsl.so.1...Reading symbols from /usr/lib/debug/lib/libnsl-2.11.1.so...done.
done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_nis.so.2...Reading symbols from /usr/lib/debug/lib/libnss_nis-2.11.1.so...done.
done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...Reading symbols from /usr/lib/debug/lib/libnss_files-2.11.1.so...done.
done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /usr/lib/postgresql/8.4/lib/pgq_triggers.so...Reading symbols from /usr/lib/debug/usr/lib/postgresql/8.4/lib/pgq_triggers.so...done.
done.
Loaded symbols for /usr/lib/postgresql/8.4/lib/pgq_triggers.so
Reading symbols from /usr/lib/postgresql/8.4/lib/plpgsql.so...Reading symbols from /usr/lib/debug/usr/lib/postgresql/8.4/lib/plpgsql.so...done.
done.
Loaded symbols for /usr/lib/postgresql/8.4/lib/plpgsql.so
Reading symbols from /usr/lib/postgresql/8.4/lib/pgq_lowlevel.so...Reading symbols from /usr/lib/debug/usr/lib/postgresql/8.4/lib/pgq_lowlevel.so...done.
done.
Loaded symbols for /usr/lib/postgresql/8.4/lib/pgq_lowlevel.so
Core was generated by `postgres: rails database_prod 127.0.0.1(40408) INSERT '.
Program terminated with signal 11, Segmentation fault.
#0 pfree (pointer=0x0) at mcxt.c:591
591 mcxt.c: No such file or directory.
        in mcxt.c
(gdb) bt
#0 pfree (pointer=0x0) at mcxt.c:591
#1 0x00007f19791f6d8e in free_info (arg=<value optimized out>, relid=17402) at common.c:244
#2 relcache_reset_cb (arg=<value optimized out>, relid=17402) at common.c:257
#3 0x00007f1a84f700af in LocalExecuteInvalidationMessage (msg=0x7f1a866fa2d0) at inval.c:516
#4 0x00007f1a84f6f902 in ProcessInvalidationMessages (hdr=<value optimized out>, func=0x7f1a84f6ff70 <LocalExecuteInvalidationMessage>) at inval.c:397
#5 0x00007f1a84f6f940 in CommandEndInvalidationMessages () at inval.c:1006
#6 0x00007f1a84d4f05e in AtCommit_LocalCache () at xact.c:1031
#7 CommandCounterIncrement () at xact.c:656
#8 0x00007f1a84e36d5d in _SPI_execute_plan (plan=<value optimized out>, paramLI=<value optimized out>, snapshot=<value optimized out>, crosscheck_snapshot=<value optimized out>, read_only=0 '\000', fire_triggers=<value optimized out>, tcount=0) at spi.c:1776
#9 0x00007f1a84e374db in SPI_execute_plan (plan=0x7f1a866814d0, Values=<value optimized out>, Nulls=<value optimized out>, read_only=0 '\000', tcount=0) at spi.c:392
#10 0x00007f19791f721d in fill_tbl_info (rel=<value optimized out>) at common.c:218
#11 pgq_find_table_info (rel=<value optimized out>) at common.c:276
#12 0x00007f19791f74d6 in pgq_prepare_event (ev=0x7fff9587acf0, tg=0x7fff9587b240, newstyle=0 '\000') at common.c:373
#13 0x00007f19791f7942 in pgq_logtriga (fcinfo=0x7fff9587adc0) at logtriga.c:68
#14 0x00007f1a84dfa1e7 in ExecCallTriggerFunc (trigdata=<value optimized out>, tgindx=<value optimized out>, finfo=<value optimized out>, instr=0x0, per_tuple_context=<value optimized out>) at trigger.c:1550
#15 0x00007f1a84dfa6b8 in AfterTriggerExecute (events=<value optimized out>, firing_id=<value optimized out>, estate=<value optimized out>, delete_ok=<value optimized out>) at trigger.c:2771
#16 afterTriggerInvokeEvents (events=<value optimized out>, firing_id=<value optimized out>, estate=<value optimized out>, delete_ok=<value optimized out>) at trigger.c:2950
#17 0x00007f1a84dfb63a in AfterTriggerEndQuery (estate=0x7f1a866b06a0) at trigger.c:3140
#18 0x00007f1a84ecb905 in ProcessQuery (plan=0x7f1a866c2ec8, sourceText=0x7f1a866189c0 "INSERT INTO \"payments\" (\"geo\", \"product_type\", \"updated_on\", \"payment_method\", \"experimental_cohort\", \"user_id\", \"recipient_id\", \"created_on\", \"user_agent\", \"subscription_id\", \"referrer\", \"ip\", \"subsc"..., params=0x0, dest=<value optimized out>, completionTag=0x7fff9587b3a0 "INSERT 0 1") at pquery.c:233
#19 0x00007f1a84ecbad8 in PortalRunMulti (portal=0x7f1a86655150, isTopLevel=1 '\001', dest=<value optimized out>, altdest=0x7f1a8532bd00, completionTag=0x7fff9587b3a0 "INSERT 0 1") at pquery.c:1269
#20 0x00007f1a84ecbd7c in FillPortalStore (portal=0x7f1a86655150, isTopLevel=0 '\000') at pquery.c:1061
#21 0x00007f1a84ecc41d in PortalRun (portal=<value optimized out>, count=<value optimized out>, isTopLevel=0 '\000', dest=<value optimized out>, altdest=<value optimized out>, completionTag=<value optimized out>) at pquery.c:802
#22 0x00007f1a84ec7201 in exec_simple_query (query_string=<value optimized out>) at postgres.c:997
#23 0x00007f1a84ec8a48 in PostgresMain (argc=<value optimized out>, argv=<value optimized out>, username=<value optimized out>) at postgres.c:3681
#24 0x00007f1a84e9adf1 in BackendRun () at postmaster.c:3467
#25 BackendStartup () at postmaster.c:3081
#26 ServerLoop () at postmaster.c:1387
#27 0x00007f1a84e9d541 in PostmasterMain (argc=-2041090048, argv=0x7f1a86537160) at postmaster.c:1040
#28 0x00007f1a84e45a43 in main (argc=5, argv=0x7f1a86537140) at main.c:188
(gdb)

[LoOoD (gman)]

postgres segfaulted again. Here is the backtrace from the second coredump.

puppet:~$ gdb -q -c core.db8.2-2-12 /usr/lib/postgresql/8.4/bin/postgres
Reading symbols from /usr/lib/postgresql/8.4/bin/postgres...Reading symbols from /usr/lib/debug/usr/lib/postgresql/8.4/bin/postgres...done.
done.
[New Thread 23719]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /usr/lib/libxml2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/libpam.so.0...Reading symbols from /usr/lib/debug/lib/libpam.so.0.82.2...done.
done.
Loaded symbols for /lib/libpam.so.0
Reading symbols from /lib/libssl.so.0.9.8...Reading symbols from /usr/lib/debug/lib/libssl.so.0.9.8...done.
done.
Loaded symbols for /lib/libssl.so.0.9.8
Reading symbols from /lib/libcrypto.so.0.9.8...Reading symbols from /usr/lib/debug/lib/libcrypto.so.0.9.8...done.
done.
Loaded symbols for /lib/libcrypto.so.0.9.8
Reading symbols from /usr/lib/libkrb5.so.3...Reading symbols from /usr/lib/debug/usr/lib/libkrb5.so.3.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /lib/libdl.so.2...Reading symbols from /usr/lib/debug/lib/libdl-2.11.1.so...done.
done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.so.6...Reading symbols from /usr/lib/debug/lib/libm-2.11.1.so...done.
done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /lib/libc.so.6...Reading symbols from /usr/lib/debug/lib/libc-2.11.1.so...done.
done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/libcrypt.so.1...Reading symbols from /usr/lib/debug/lib/libcrypt-2.11.1.so...done.
done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /lib/libresolv.so.2...Reading symbols from /usr/lib/debug/lib/libresolv-2.11.1.so...done.
done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib/ld-2.11.1.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbo...

[LoOoD (gman)]

And another

puppet:~$ gdb -q -c core.2-6-12-00 /usr/lib/postgresql/8.4/bin/postgres
Reading symbols from /usr/lib/postgresql/8.4/bin/postgres...Reading symbols from /usr/lib/debug/usr/lib/postgresql/8.4/bin/postgres...done.
done.
[New Thread 14948]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /usr/lib/libxml2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/libpam.so.0...Reading symbols from /usr/lib/debug/lib/libpam.so.0.82.2...done.
done.
Loaded symbols for /lib/libpam.so.0
Reading symbols from /lib/libssl.so.0.9.8...Reading symbols from /usr/lib/debug/lib/libssl.so.0.9.8...done.
done.
Loaded symbols for /lib/libssl.so.0.9.8
Reading symbols from /lib/libcrypto.so.0.9.8...Reading symbols from /usr/lib/debug/lib/libcrypto.so.0.9.8...done.
done.
Loaded symbols for /lib/libcrypto.so.0.9.8
Reading symbols from /usr/lib/libkrb5.so.3...Reading symbols from /usr/lib/debug/usr/lib/libkrb5.so.3.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /lib/libdl.so.2...Reading symbols from /usr/lib/debug/lib/libdl-2.11.1.so...done.
done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.so.6...Reading symbols from /usr/lib/debug/lib/libm-2.11.1.so...done.
done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /lib/libc.so.6...Reading symbols from /usr/lib/debug/lib/libc-2.11.1.so...done.
done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/libcrypt.so.1...Reading symbols from /usr/lib/debug/lib/libcrypt-2.11.1.so...done.
done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /lib/libresolv.so.2...Reading symbols from /usr/lib/debug/lib/libresolv-2.11.1.so...done.
done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib/ld-2.11.1.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/lib...