Use GCC 4.4 for RHEL 5 | Bitmap tests crash the 5.5 server on Jenkins centos5-32 release

[Laurynas Biveinis]

http://jenkins.percona.com/job/percona-server-5.5-trunk/357/BUILD_TYPE=release,Host=centos5-32/consoleText

All crashes have stacktraces with the following stack top:

Program terminated with signal 11, Segmentation fault.
#0 0xb7788420 in __kernel_vsyscall ()
#0 0xb7788420 in __kernel_vsyscall ()
#1 0xb7751ce7 in pthread_kill () from /lib/libpthread.so.0
#2 0x08441adb in my_write_core (sig=11)
    at /home/jenkins/workspace/percona-server-5.5-trunk/BUILD_TYPE/release/Host/centos5-32/Percona-Server/mysys/stacktrace.c:433
#3 0x082eae90 in handle_fatal_signal (sig=11)
    at /home/jenkins/workspace/percona-server-5.5-trunk/BUILD_TYPE/release/Host/centos5-32/Percona-Server/sql/signal_handler.cc:250
#4 <signal handler called>
#5 log_max_modified_age_async ()
    at /home/jenkins/workspace/percona-server-5.5-trunk/BUILD_TYPE/release/Host/centos5-32/Percona-Server/storage/innobase/log/log0log.c:431
#6 log_close ()
    at /home/jenkins/workspace/percona-server-5.5-trunk/BUILD_TYPE/release/Host/centos5-32/Percona-Server/storage/innobase/log/log0log.c:539
#7 0x085c67b7 in mtr_log_reserve_and_write (mtr=0xabfd9950)
    at /home/jenkins/workspace/percona-server-5.5-trunk/BUILD_TYPE/release/Host/centos5-32/Percona-Server/storage/innobase/mtr/mtr0mtr.c:255
#8 mtr_commit (mtr=0xabfd9950)
    at /home/jenkins/workspace/percona-server-5.5-trunk/BUILD_TYPE/release/Host/centos5-32/Percona-Server/storage/innobase/mtr/mtr0mtr.c:302

[Sergei Glushchenko (sergei.glushchenko)]

The bug caused by http://gcc.gnu.org/bugzilla/show_bug.cgi?id=51332 in GCC 4.1. Bitmap code use log_get_tracked_lsn which on 32bit CentOS with i686 falls to buggy __sync_add_and_fetch.

Workaround is to compile PS using gcc 4.4 on CentOS 5 or not define HAVE_ATOMIC_BUILTINS_64 for gcc 4.1 and fall back to using mutex.

[Raghavendra D Prabhu (raghavendra-prabhu)]

We have had the following logic in earlier builds, it may have been not added in new builds, will be good to use gcc44 for rhel5 again.

# Use modern gcc for rhel5.
if test "x$REDHAT_VERSION" == "xrhel5"
then
    export CC=gcc44 CXX=g++44
fi

[Sergei Glushchenko (sergei.glushchenko)]

Hi Raghu,

I agree. Apart from Jenkins we should also make this change in our build scripts.

[Laurynas Biveinis (laurynas-biveinis)]

Opening for 5.6 as well. Might be a different issue, but logging here for now.

http://jenkins.percona.com/job/percona-server-5.6-trunk/297/BUILD_TYPE=release,Host=centos5-32/console

00:27:48.073 innodb.percona_changed_pages w2 [ fail ]

00:28:00.379 Program terminated with signal 11, Segmentation fault.
00:28:00.379 #0 0xb76f8405 in __kernel_vsyscall ()
00:28:00.379 #0 0xb76f8405 in __kernel_vsyscall ()
00:28:00.379 #1 0xb76ddce7 in pthread_kill () from /lib/libpthread.so.0
00:28:00.379 #2 0x0857853b in my_write_core (sig=11)
00:28:00.379 at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/mysys/stacktrace.c:422
00:28:00.379 #3 0x08290d73 in handle_fatal_signal (sig=11)
00:28:00.379 at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/signal_handler.cc:254
00:28:00.379 #4 <signal handler called>
00:28:00.379 #5 0x08609fbb in i_s_innodb_changed_pages_fill (thd=0x9b0cfb0,
00:28:00.379 tables=0x9bca060, cond=0x0)
00:28:00.379 at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/storage/innobase/handler/i_s.cc:8456
00:28:00.379 #6 0x083560fc in get_schema_tables_result (join=0x9bca858,
00:28:00.379 executed_place=PROCESSED_BY_JOIN_EXEC)
00:28:00.379 at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/sql_show.cc:7799
00:28:00.379 #7 0x08347cf1 in JOIN::prepare_result (this=0x9bca858,
00:28:00.379 columns_list=0xa91bccf4)
00:28:00.379 at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/sql_select.cc:823
00:28:00.379 #8 0x082fafa6 in JOIN::exec (this=0x9bca858)
00:28:00.379 at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/sql_executor.cc:116
00:28:00.379 #9 0x0834af0d in mysql_execute_select (thd=0x9b0cfb0, select_lex=0x9b0ee5c,
00:28:00.379 free_join=true)
00:28:00.379 at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/sql_select.cc:1101
00:28:00.379 #10 0x0834b2ad in mysql_select (thd=0x9b0cfb0, tables=0x9bca060, wild_num=0,
00:28:00.379 fields=..., conds=0x0, order=0x9b0ef50, group=0x9b0eeec, having=0x0,
00:28:00.379 select_options=2684619520, result=0x9bca840, unit=0x9b0e9f8,
00:28:00.379 select_lex=0x9b0ee5c)
00:28:00.379 at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/sql_select.cc:1222
00:28:00.379 #11 0x0834b50d in handle_select (thd=0x9b0cfb0, result=0x9bca840,
00:28:00.379 setup_tables_done_option=0)
00:28:00.379 at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/sql_select.cc:111
00:28:00.379 #12 0x0831e3cb in execute_sqlcom_select (thd=0x9b0cfb0, all_tables=0x9bca060)
00:28:00.379 at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/sql_parse.cc:5570
00:28:00.379 #13 0x08325c7c in mysql_execute_command (thd=0x9b0cfb0)
00:28:00.379 at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/sql_parse.cc:2955
00:28:00.37...

[Laurynas Biveinis (laurynas-biveinis)]

After the 5.6.22 merge CentOS 5 release build started crashing on the MTR bootstrap shutdown:

Program received signal SIGSEGV, Segmentation fault.
logs_empty_and_mark_files_at_shutdown () at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/storage/innobase/log/log0log.cc:3597
3597 lsn = log_sys->lsn;
(gdb) bt
#0 logs_empty_and_mark_files_at_shutdown () at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/storage/innobase/log/log0log.cc:3597
#1 0x0871f44a in innobase_shutdown_for_mysql () at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/storage/innobase/srv/srv0start.cc:2980
#2 0x08645279 in innobase_end (hton=0x8d5f610, type=HA_PANIC_CLOSE) at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/storage/innobase/handler/ha_innodb.cc:3809
#3 0x081a2fee in ha_finalize_handlerton (plugin=0x8da68f8) at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/handler.cc:641
#4 0x08331bb7 in plugin_deinitialize (plugin=0x8da68f8, ref_check=true) at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/sql_plugin.cc:944
#5 0x08336e7f in reap_plugins () at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/sql_plugin.cc:1023
#6 0x0833833a in plugin_shutdown () at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/sql_plugin.cc:1725
#7 0x08197233 in clean_up (print_message=true) at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/mysqld.cc:1935
#8 0x08197975 in unireg_abort (exit_code=2) at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/mysqld.cc:1821
#9 0x0819b728 in init_server_components () at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/mysqld.cc:4992
#10 0x0819bc7d in mysqld_main (argc=19, argv=0x8d5ee68) at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/mysqld.cc:5611
#11 0x0818e4d2 in main (argc=Cannot access memory at address 0x0
) at /home/jenkins/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/centos5-32/sql/main.cc:25

Disassembly suggests that http://gcc.gnu.org/bugzilla/show_bug.cgi?id=51332 might be relevant here too:

(gdb) x/i $pc
0x86943ec <logs_empty_and_mark_files_at_shutdown()+924>: mov 0x113694(%ebx),%esi

0x086943de <logs_empty_and_mark_files_at_shutdown()+910>: xchg %eax,%ebx
0x086943df <logs_empty_and_mark_files_at_shutdown()+911>: lock cmpxchg8b (%esi)
0x086943e3 <logs_empty_and_mark_files_at_shutdown()+915>: xchg %eax,%ebx
0x086943e4 <logs_empty_and_mark_files_at_shutdown()+916>: mov %eax,-0x38(%ebp)
0x086943e7 <logs_empty_and_mark_files_at_shutdown()+919>: mov %edx,-0x34(%ebp)
0x086943ea <logs_empty_and_mark_files_at_shutdown()+922>: jne 0x86943cd <logs_empty_and_mark_files_at_shutdown()+893>
0x086943ec <logs_empty_and_mark_files_at_shutdown()+924>: mov 0x113694(%ebx),%esi

I haven't worked out the above for a 100% confirmation, but I believe we must ...

[Tomislav Plavcic (tplavcic)]

Made a change in jenkins jobs so that on centos5 GCC 4.4 is used.
Laurynas thinks maybe we should add buggy gcc version to cmake so will not close it right now until that is clear.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3056