postgres segfaulted again. Here is the backtrace from the second coredump.
puppet:~$ gdb -q -c core.db8.2-2-12 /usr/lib/postgresql/8.4/bin/postgres
Reading symbols from /usr/lib/postgresql/8.4/bin/postgres...Reading symbols from /usr/lib/debug/usr/lib/postgresql/8.4/bin/postgres...done.
done.
[New Thread 23719]
warning: Can't read pathname for load map: Input/output error.
Reading symbols from /usr/lib/libxml2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/libpam.so.0...Reading symbols from /usr/lib/debug/lib/libpam.so.0.82.2...done.
done.
Loaded symbols for /lib/libpam.so.0
Reading symbols from /lib/libssl.so.0.9.8...Reading symbols from /usr/lib/debug/lib/libssl.so.0.9.8...done.
done.
Loaded symbols for /lib/libssl.so.0.9.8
Reading symbols from /lib/libcrypto.so.0.9.8...Reading symbols from /usr/lib/debug/lib/libcrypto.so.0.9.8...done.
done.
Loaded symbols for /lib/libcrypto.so.0.9.8
Reading symbols from /usr/lib/libkrb5.so.3...Reading symbols from /usr/lib/debug/usr/lib/libkrb5.so.3.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /lib/libdl.so.2...Reading symbols from /usr/lib/debug/lib/libdl-2.11.1.so...done.
done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.so.6...Reading symbols from /usr/lib/debug/lib/libm-2.11.1.so...done.
done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /lib/libc.so.6...Reading symbols from /usr/lib/debug/lib/libc-2.11.1.so...done.
done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/libcrypt.so.1...Reading symbols from /usr/lib/debug/lib/libcrypt-2.11.1.so...done.
done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /lib/libresolv.so.2...Reading symbols from /usr/lib/debug/lib/libresolv-2.11.1.so...done.
done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib/ld-2.11.1.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/libsasl2.so.2...Reading symbols from /usr/lib/debug/usr/lib/libsasl2.so.2.0.23...done.
done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libgnutls.so.26...Reading symbols from /usr/lib/debug/usr/lib/libgnutls.so.26.14.12...done.
done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /usr/lib/libtasn1.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /lib/libgcrypt.so.11...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcrypt.so.11
Reading symbols from /lib/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libgpg-error.so.0
Reading symbols from /lib/libnss_compat.so.2...Reading symbols from /usr/lib/debug/lib/libnss_compat-2.11.1.so...done.
done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnsl.so.1...Reading symbols from /usr/lib/debug/lib/libnsl-2.11.1.so...done.
done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_nis.so.2...Reading symbols from /usr/lib/debug/lib/libnss_nis-2.11.1.so...done.
done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...Reading symbols from /usr/lib/debug/lib/libnss_files-2.11.1.so...done.
done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /usr/lib/postgresql/8.4/lib/pgq_triggers.so...Reading symbols from /usr/lib/debug/usr/lib/postgresql/8.4/lib/pgq_triggers.so...done.
done.
Loaded symbols for /usr/lib/postgresql/8.4/lib/pgq_triggers.so
Reading symbols from /usr/lib/postgresql/8.4/lib/plpgsql.so...Reading symbols from /usr/lib/debug/usr/lib/postgresql/8.4/lib/plpgsql.so...done.
done.
Loaded symbols for /usr/lib/postgresql/8.4/lib/plpgsql.so
Reading symbols from /usr/lib/postgresql/8.4/lib/pgq_lowlevel.so...Reading symbols from /usr/lib/debug/usr/lib/postgresql/8.4/lib/pgq_lowlevel.so...done.
done.
Loaded symbols for /usr/lib/postgresql/8.4/lib/pgq_lowlevel.so
Core was generated by `postgres: rails database_prod 127.0.0.1(2207) UPDATE '.
Program terminated with signal 11, Segmentation fault.
#0 pfree (pointer=0x0) at mcxt.c:591
591 mcxt.c: No such file or directory.
in mcxt.c
(gdb) bt
#0 pfree (pointer=0x0) at mcxt.c:591
#1 0x00007f19791f6d8e in free_info (arg=<value optimized out>, relid=4385522) at common.c:244
#2 relcache_reset_cb (arg=<value optimized out>, relid=4385522) at common.c:257
#3 0x00007f1a84f700af in LocalExecuteInvalidationMessage (msg=0x7f1a86760280) at inval.c:516
#4 0x00007f1a84f6f902 in ProcessInvalidationMessages (hdr=<value optimized out>, func=0x7f1a84f6ff70 <LocalExecuteInvalidationMessage>) at inval.c:397
#5 0x00007f1a84f6f940 in CommandEndInvalidationMessages () at inval.c:1006
#6 0x00007f1a84d4f05e in AtCommit_LocalCache () at xact.c:1031
#7 CommandCounterIncrement () at xact.c:656
#8 0x00007f1a84e36d5d in _SPI_execute_plan (plan=<value optimized out>, paramLI=<value optimized out>, snapshot=<value optimized out>, crosscheck_snapshot=<value optimized out>, read_only=0 '\000', fire_triggers=<value optimized out>, tcount=0) at spi.c:1776
#9 0x00007f1a84e374db in SPI_execute_plan (plan=0x7f1a86697740, Values=<value optimized out>, Nulls=<value optimized out>, read_only=0 '\000', tcount=0) at spi.c:392
#10 0x00007f19791f721d in fill_tbl_info (rel=<value optimized out>) at common.c:218
#11 pgq_find_table_info (rel=<value optimized out>) at common.c:276
#12 0x00007f19791f74d6 in pgq_prepare_event (ev=0x7fff9587ad60, tg=0x7fff9587b2b0, newstyle=0 '\000') at common.c:373
#13 0x00007f19791f7942 in pgq_logtriga (fcinfo=0x7fff9587ae30) at logtriga.c:68
#14 0x00007f1a84dfa1e7 in ExecCallTriggerFunc (trigdata=<value optimized out>, tgindx=<value optimized out>, finfo=<value optimized out>, instr=0x0, per_tuple_context=<value optimized out>) at trigger.c:1550
#15 0x00007f1a84dfa6b8 in AfterTriggerExecute (events=<value optimized out>, firing_id=<value optimized out>, estate=<value optimized out>, delete_ok=<value optimized out>) at trigger.c:2771
#16 afterTriggerInvokeEvents (events=<value optimized out>, firing_id=<value optimized out>, estate=<value optimized out>, delete_ok=<value optimized out>) at trigger.c:2950
#17 0x00007f1a84dfb63a in AfterTriggerEndQuery (estate=0x7f1a86742a30) at trigger.c:3140
#18 0x00007f1a84ecb905 in ProcessQuery (plan=0x7f1a86758290, sourceText=0x7f1a8661a400 "UPDATE \"emoticons\" SET global = true", params=0x0, dest=<value optimized out>, completionTag=0x7fff9587b5b0 "UPDATE 52") at pquery.c:233
#19 0x00007f1a84ecbad8 in PortalRunMulti (portal=0x7f1a86655b60, isTopLevel=1 '\001', dest=<value optimized out>, altdest=0x7f1a86708800, completionTag=0x7fff9587b5b0 "UPDATE 52") at pquery.c:1269
#20 0x00007f1a84ecc382 in PortalRun (portal=<value optimized out>, count=<value optimized out>, isTopLevel=0 '\000', dest=<value optimized out>, altdest=<value optimized out>, completionTag=<value optimized out>) at pquery.c:823
#21 0x00007f1a84ec7201 in exec_simple_query (query_string=<value optimized out>) at postgres.c:997
#22 0x00007f1a84ec8a48 in PostgresMain (argc=<value optimized out>, argv=<value optimized out>, username=<value optimized out>) at postgres.c:3681
#23 0x00007f1a84e9adf1 in BackendRun () at postmaster.c:3467
#24 BackendStartup () at postmaster.c:3081
#25 ServerLoop () at postmaster.c:1387
#26 0x00007f1a84e9d541 in PostmasterMain (argc=-2041090048, argv=0x7f1a86537160) at postmaster.c:1040
#27 0x00007f1a84e45a43 in main (argc=5, argv=0x7f1a86537140) at main.c:188
(gdb)
postgres segfaulted again. Here is the backtrace from the second coredump.
puppet:~$ gdb -q -c core.db8.2-2-12 /usr/lib/ postgresql/ 8.4/bin/ postgres postgresql/ 8.4/bin/ postgres. ..Reading symbols from /usr/lib/ debug/usr/ lib/postgresql/ 8.4/bin/ postgres. ..done.
Reading symbols from /usr/lib/
done.
[New Thread 23719]
warning: Can't read pathname for load map: Input/output error. libxml2. so.2... (no debugging symbols found)...done. libxml2. so.2 so.0... Reading symbols from /usr/lib/ debug/lib/ libpam. so.0.82. 2...done. so.0.9. 8...Reading symbols from /usr/lib/ debug/lib/ libssl. so.0.9. 8...done. so.0.9. 8 so.0.9. 8...Reading symbols from /usr/lib/ debug/lib/ libcrypto. so.0.9. 8...done. so.0.9. 8 libkrb5. so.3... Reading symbols from /usr/lib/ debug/usr/ lib/libkrb5. so.3.3. ..(no debugging symbols found)...done. libkrb5. so.3 err.so. 2...(no debugging symbols found)...done. err.so. 2 libgssapi_ krb5.so. 2...(no debugging symbols found)...done. libgssapi_ krb5.so. 2 so.2... Reading symbols from /usr/lib/ debug/lib/ libdl-2. 11.1.so. ..done. so.6... Reading symbols from /usr/lib/ debug/lib/ libm-2. 11.1.so. ..done. libldap_ r-2.4.so. 2...(no debugging symbols found)...done. libldap_ r-2.4.so. 2 so.6... Reading symbols from /usr/lib/ debug/lib/ libc-2. 11.1.so. ..done. so.1... (no debugging symbols found)...done. so.1... Reading symbols from /usr/lib/ debug/lib/ libcrypt- 2.11.1. so...done. libk5crypto. so.3... (no debugging symbols found)...done. libk5crypto. so.3 libkrb5support. so.0... (no debugging symbols found)...done. libkrb5support. so.0 s.so.1. ..(no debugging symbols found)...done. s.so.1 so.2... Reading symbols from /usr/lib/ debug/lib/ libresolv- 2.11.1. so...done. .so.0.. .(no debugging symbols found)...done. .so.0 ld-linux- x86-64. so.2... Reading symbols from /usr/lib/ debug/lib/ ld-2.11. 1.so... done. ld-linux- x86-64. so.2 liblber- 2.4.so. 2...(no debugging symbols found)...done. liblber- 2.4.so. 2 libsasl2. so.2... Reading symbols from /usr/lib/ debug/usr/ lib/libsasl2. so.2.0. 23...done. libsasl2. so.2 libgnutls. so.26.. .Reading symbols from /usr/lib/ debug/usr/ lib/libgnutls. so.26.14. 12...done. libgnutls. so.26 libtasn1. so.3... (no debugging symbols found)...done. libtasn1. so.3 so.11.. .(no debugging symbols found)...done. so.11 error.so. 0...(no debugging symbols found)...done. error.so. 0 compat. so.2... Reading symbols from /usr/lib/ debug/lib/ libnss_ compat- 2.11.1. so...done. compat. so.2 so.1... Reading symbols from /usr/lib/ debug/lib/ libnsl- 2.11.1. so...done. nis.so. 2...Reading symbols from /usr/lib/ debug/lib/ libnss_ nis-2.11. 1.so... done. nis.so. 2 files.so. 2...Reading symbols from /usr/lib/ debug/lib/ libnss_ files-2. 11.1.so. ..done. files.so. 2 postgresql/ 8.4/lib/ pgq_triggers. so...Reading symbols from /usr/lib/ debug/usr/ lib/postgresql/ 8.4/lib/ pgq_triggers. so...done. postgresql/ 8.4/lib/ pgq_triggers. so postgresql/ 8.4/lib/ plpgsql. so...Reading symbols from /usr/lib/ debug/usr/ lib/postgresql/ 8.4/lib/ plpgsql. so...done. postgresql/ 8.4/lib/ plpgsql. so postgresql/ 8.4/lib/ pgq_lowlevel. so...Reading symbols from /usr/lib/ debug/usr/ lib/postgresql/ 8.4/lib/ pgq_lowlevel. so...done. postgresql/ 8.4/lib/ pgq_lowlevel. so alidationMessag e (msg=0x7f1a8676 0280) at inval.c:516 tionMessages (hdr=<value optimized out>, func=0x7f1a84f6ff70 <LocalExecuteIn validationMessa ge>) at inval.c:397 idationMessages () at inval.c:1006 ncrement () at xact.c:656 snapshot= <value optimized out>, read_only=0 '\000', fire_triggers= <value optimized out>, tcount=0) at spi.c:1776 97740, Values=<value optimized out>, Nulls=<value optimized out>, read_only=0 '\000', tcount=0) at spi.c:392 0x7fff9587ae30) at logtriga.c:68 context= <value optimized out>) at trigger.c:1550 okeEvents (events=<value optimized out>, firing_id=<value optimized out>, estate=<value optimized out>, delete_ok=<value optimized out>) at trigger.c:2950 Query (estate= 0x7f1a86742a30) at trigger.c:3140 58290, sourceText= 0x7f1a8661a400 "UPDATE \"emoticons\" SET global = true", params=0x0, dest=<value optimized out>, completionTag= 0x7fff9587b5b0 "UPDATE 52") at pquery.c:233 0x7f1a86655b60, isTopLevel=1 '\001', dest=<value optimized out>, altdest= 0x7f1a86708800, completionTag= 0x7fff9587b5b0 "UPDATE 52") at pquery.c:1269 <value optimized out>) at pquery.c:823 string= <value optimized out>) at postgres.c:997 7160) at postmaster.c:1040 7140) at main.c:188
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /lib/libpam.
done.
Loaded symbols for /lib/libpam.so.0
Reading symbols from /lib/libssl.
done.
Loaded symbols for /lib/libssl.
Reading symbols from /lib/libcrypto.
done.
Loaded symbols for /lib/libcrypto.
Reading symbols from /usr/lib/
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/
Reading symbols from /lib/libcom_
Loaded symbols for /lib/libcom_
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /lib/libdl.
done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.
done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /lib/libc.
done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libz.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/libcrypt.
done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /lib/libkeyutil
Loaded symbols for /lib/libkeyutil
Reading symbols from /lib/libresolv.
done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libpthread
Loaded symbols for /lib/libpthread
Reading symbols from /lib64/
done.
Loaded symbols for /lib64/
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /usr/lib/
done.
Loaded symbols for /usr/lib/
Reading symbols from /usr/lib/
done.
Loaded symbols for /usr/lib/
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /lib/libgcrypt.
Loaded symbols for /lib/libgcrypt.
Reading symbols from /lib/libgpg-
Loaded symbols for /lib/libgpg-
Reading symbols from /lib/libnss_
done.
Loaded symbols for /lib/libnss_
Reading symbols from /lib/libnsl.
done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_
done.
Loaded symbols for /lib/libnss_
Reading symbols from /lib/libnss_
done.
Loaded symbols for /lib/libnss_
Reading symbols from /usr/lib/
done.
Loaded symbols for /usr/lib/
Reading symbols from /usr/lib/
done.
Loaded symbols for /usr/lib/
Reading symbols from /usr/lib/
done.
Loaded symbols for /usr/lib/
Core was generated by `postgres: rails database_prod 127.0.0.1(2207) UPDATE '.
Program terminated with signal 11, Segmentation fault.
#0 pfree (pointer=0x0) at mcxt.c:591
591 mcxt.c: No such file or directory.
in mcxt.c
(gdb) bt
#0 pfree (pointer=0x0) at mcxt.c:591
#1 0x00007f19791f6d8e in free_info (arg=<value optimized out>, relid=4385522) at common.c:244
#2 relcache_reset_cb (arg=<value optimized out>, relid=4385522) at common.c:257
#3 0x00007f1a84f700af in LocalExecuteInv
#4 0x00007f1a84f6f902 in ProcessInvalida
#5 0x00007f1a84f6f940 in CommandEndInval
#6 0x00007f1a84d4f05e in AtCommit_LocalCache () at xact.c:1031
#7 CommandCounterI
#8 0x00007f1a84e36d5d in _SPI_execute_plan (plan=<value optimized out>, paramLI=<value optimized out>, snapshot=<value optimized out>, crosscheck_
#9 0x00007f1a84e374db in SPI_execute_plan (plan=0x7f1a866
#10 0x00007f19791f721d in fill_tbl_info (rel=<value optimized out>) at common.c:218
#11 pgq_find_table_info (rel=<value optimized out>) at common.c:276
#12 0x00007f19791f74d6 in pgq_prepare_event (ev=0x7fff9587ad60, tg=0x7fff9587b2b0, newstyle=0 '\000') at common.c:373
#13 0x00007f19791f7942 in pgq_logtriga (fcinfo=
#14 0x00007f1a84dfa1e7 in ExecCallTriggerFunc (trigdata=<value optimized out>, tgindx=<value optimized out>, finfo=<value optimized out>, instr=0x0, per_tuple_
#15 0x00007f1a84dfa6b8 in AfterTriggerExecute (events=<value optimized out>, firing_id=<value optimized out>, estate=<value optimized out>, delete_ok=<value optimized out>) at trigger.c:2771
#16 afterTriggerInv
#17 0x00007f1a84dfb63a in AfterTriggerEnd
#18 0x00007f1a84ecb905 in ProcessQuery (plan=0x7f1a867
#19 0x00007f1a84ecbad8 in PortalRunMulti (portal=
#20 0x00007f1a84ecc382 in PortalRun (portal=<value optimized out>, count=<value optimized out>, isTopLevel=0 '\000', dest=<value optimized out>, altdest=<value optimized out>, completionTag=
#21 0x00007f1a84ec7201 in exec_simple_query (query_
#22 0x00007f1a84ec8a48 in PostgresMain (argc=<value optimized out>, argv=<value optimized out>, username=<value optimized out>) at postgres.c:3681
#23 0x00007f1a84e9adf1 in BackendRun () at postmaster.c:3467
#24 BackendStartup () at postmaster.c:3081
#25 ServerLoop () at postmaster.c:1387
#26 0x00007f1a84e9d541 in PostmasterMain (argc=-2041090048, argv=0x7f1a8653
#27 0x00007f1a84e45a43 in main (argc=5, argv=0x7f1a8653
(gdb)