Reproducible. Got it again: 1) wsrep_provider="none" in my.cnf 2) mysql restart 3) run the same commands: SET GLOBAL wsrep_provider="/usr/lib64/galera/libgalera_smm.so"; SHOW STATUS LIKE 'wsrep_%'; 4) crash 130228 20:30:44 [Note] WSREP: Passing config to GCS: base_host = 10.82.25.164; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 1G; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; pc.ignore_sb = true; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3; socket.ssl_ca = /etc/my.ssl/cacert.pem; socket.ssl_cert = /etc/my.ssl/dev-mysql-client.crt; socket.ssl_key = /etc/my.ssl/dev-mysql-client-nopwd.key 130228 20:30:44 [Note] WSREP: Assign initial position for certification: 11379177, protocol version: -1 130228 20:30:45 [Warning] WSREP: gcs_caused() returned -103 (Software caused connection abort) *** glibc detected *** /usr/sbin/mysqld: double free or corruption (out): 0x00000000019d8a60 *** ======= Backtrace: ========= /lib64/libc.so.6[0x2ab40eef46af] /lib64/libc.so.6(cfree+0x4b)[0x2ab40eef88db] /usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM10stats_freeEP15wsrep_stats_var+0x48)[0x2ab48e76d968] /usr/sbin/mysqld(_Z17wsrep_free_statusP3THD+0x28)[0x6993b8] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4d29)[0x595629] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x1b9)[0x596f39] /usr/sbin/mysqld[0x59780e] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1793)[0x599d83] /usr/sbin/mysqld(_Z10do_commandP3THD+0x119)[0x59a309] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1f3)[0x6505a3] /usr/sbin/mysqld(handle_one_connection+0x4c)[0x65075c] /lib64/libpthread.so.0[0x2ab40de8e83d] /lib64/libc.so.6(clone+0x6d)[0x2ab40ef5803d] ======= Memory map: ======== 00400000-00f4a000 r-xp 00000000 fd:00 603309 /usr/sbin/mysqld 01149000-01195000 rw-p 00b49000 fd:00 603309 /usr/sbin/mysqld 01195000-019de000 rw-p 01195000 00:00 0 01b94000-01c18000 rw-p 00b94000 fd:00 603309 /usr/sbin/mysqld 0fe49000-15bcf000 rw-p 0fe49000 00:00 0 [heap] 30a7400000-30a7540000 r-xp 00000000 fd:00 2287041 /lib64/libcrypto.so.6 30a7540000-30a7740000 ---p 00140000 fd:00 2287041 /lib64/libcrypto.so.6 30a7740000-30a7762000 rw-p 00140000 fd:00 2287041 /lib64/libcrypto.so.6 30a7762000-30a7766000 rw-p 30a7762000 00:00 0 30a7800000-30a784f000 r-xp 00000000 fd:00 2287042 /lib64/libssl.so.0.9.8x 30a784f000-30a7a4f000 ---p 0004f000 fd:00 2287042 /lib64/libssl.so.0.9.8x 30a7a4f000-30a7a55000 rw-p 0004f000 fd:00 2287042 /lib64/libssl.so.0.9.8x 3b8b000000-3b8b015000 r-xp 00000000 fd:00 1241575 /lib64/libselinux.so.1 3b8b015000-3b8b215000 ---p 00015000 fd:00 1241575 /lib64/libselinux.so.1 3b8b215000-3b8b217000 rw-p 00015000 fd:00 1241575 /lib64/libselinux.so.1 3b8b217000-3b8b218000 rw-p 3b8b217000 00:00 0 3b8b400000-3b8b43b000 r-xp 00000000 fd:00 1241573 /lib64/libsepol.so.1 3b8b43b000-3b8b63b000 ---p 0003b000 fd:00 1241573 /lib64/libsepol.so.1 3b8b63b000-3b8b63c000 rw-p 0003b000 fd:00 1241573 /lib64/libsepol.so.1 3b8b63c000-3b8b646000 rw-p 3b8b63c000 00:00 0 3b8d400000-3b8d402000 r-xp 00000000 fd:00 1241571 /lib64/libkeyutils-1.2.so 3b8d402000-3b8d601000 ---p 00002000 fd:00 1241571 /lib64/libkeyutils-1.2.so 3b8d601000-3b8d602000 rw-p 00001000 fd:00 1241571 /lib64/libkeyutils-1.2.so 3b8dc00000-3b8dc24000 r-xp 00000000 fd:00 605969 /usr/lib64/libk5crypto.so.3.1 3b8dc24000-3b8de23000 ---p 00024000 fd:00 605969 /usr/lib64/libk5crypto.so.3.1 3b8de23000-3b8de25000 rw-p 00023000 fd:00 605969 /usr/lib64/libk5crypto.so.3.1 3b8e000000-3b8e02c000 r-xp 00000000 fd:00 605971 /usr/lib64/libgssapi_krb5.so.2.2 3b8e02c000-3b8e22c000 ---p 0002c000 fd:00 605971 /usr/lib64/libgssapi_krb5.so.2.2 3b8e22c000-3b8e22e000 rw-p 0002c000 fd:00 605971 /usr/lib64/libgssapi_krb5.so.2.2 3b8e400000-3b8e408000 r-xp 00000000 fd:00 605968 /usr/lib64/libkrb5support.so.0.1 3b8e408000-3b8e607000 ---p 00008000 fd:00 605968 /usr/lib64/libkrb5support.so.0.1 3b8e607000-3b8e608000 rw-p 00007000 fd:00 605968 /usr/lib64/libkrb5support.so.0.1 3b8ec00000-3b8ec91000 r-xp 00000000 fd:00 605970 /usr/lib64/libkrb5.so.3.3 3b8ec91000-3b8ee91000 ---p 00091000 fd:00 605970 /usr/lib64/libkrb5.so.3.3 3b8ee91000-3b8ee95000 rw-p 00091000 fd:00 605970 /usr/lib64/libkrb5.so.3.3 2ab40dc6a000-2ab40dc86000 r-xp 00000000 fd:00 1895686 /lib64/ld-2.5.so 2ab40dc86000-2ab40dc88000 rw-p 2ab40dc86000 00:00 0 2ab40de86000-2ab40de87000 r--p 0001c000 fd:00 1895686 /lib64/ld-2.5.so 2ab40de87000-2ab40de88000 rw-p 0001d000 fd:00 1895686 /lib64/ld-2.5.so 2ab40de88000-2ab40de9e000 r-xp 00000000 fd:00 1241569 /lib64/libpthread-2.5.so 2ab40de9e000-2ab40e09e000 ---p 00016000 fd:00 1241569 /lib64/libpthread-2.5.so 2ab40e09e000-2ab40e09f000 r--p 00016000 fd:00 1241569 /lib64/libpthread-2.5.so 2ab40e09f000-2ab40e0a0000 rw-p 00017000 fd:00 1241569 /lib64/libpthread-2.5.so 2ab40e0a0000-2ab40e0a4000 rw-p 2ab40e0a0000 00:00 0 2ab40e0a4000-2ab40e0b8000 r-xp 00000000 fd:00 1241584 /lib64/libz.so.1.2.3 2ab40e0b8000-2ab40e2b7000 ---p 00014000 fd:00 1241584 /lib64/libz.so.1.2.3 2ab40e2b7000-2ab40e2b8000 rw-p 00013000 fd:00 1241584 /lib64/libz.so.1.2.3 2ab40e2b8000-2ab40e2bf000 r-xp 00000000 fd:00 1241872 /lib64/librt-2.5.so 2ab40e2bf000-2ab40e4bf000 ---p 00007000 fd:00 1241872 /lib64/librt-2.5.so 2ab40e4bf000-2ab40e4c0000 r--p 00007000 fd:00 1241872 /lib64/librt-2.5.so 2ab40e4c0000-2ab40e4c1000 rw-p 00008000 fd:00 1241872 /lib64/librt-2.5.so 2ab40e4c1000-2ab40e4c2000 rw-p 2ab40e4c1000 00:00 0 2ab40e4c2000-2ab40e4cb000 r-xp 00000000 fd:00 1241549 /lib64/libcrypt-2.5.so 2ab40e4cb000-2ab40e6ca000 ---p 00009000 fd:00 1241549 /lib64/libcrypt-2.5.so 2ab40e6ca000-2ab40e6cb000 r--p 00008000 fd:00 1241549 /lib64/libcrypt-2.5.so 2ab40e6cb000-2ab40e6cc000 rw-p 00009000 fd:00 1241549 /lib64/libcrypt-2.5.so 2ab40e6cc000-2ab40e6fa000 rw-p 2ab40e6cc000 00:00 0 2ab40e6fa000-2ab40e6fc000 r-xp 00000000 fd:00 1241864 /lib64/libdl-2.5.so 2ab40e6fc000-2ab40e8fc000 ---p 00002000 fd:00 1241864 /lib64/libdl-2.5.so 2ab40e8fc000-2ab40e8fd000 r--p 00002000 fd:00 1241864 /lib64/libdl-2.5.so 2ab40e8fd000-2ab40e8fe000 rw-p 00003000 fd:00 1241864 /lib64/libdl-2.5.so 2ab40e8fe000-2ab40e8ff000 rw-p 2ab40e8fe000 00:00 0 2ab40e8ff000-2ab40e9e5000 r-xp 00000000 fd:00 595078 /usr/lib64/libstdc++.so.6.0.8 2ab40e9e5000-2ab40ebe4000 ---p 000e6000 fd:00 595078 /usr/lib64/libstdc++.so.6.0.8 2ab40ebe4000-2ab40ebea000 r--p 000e5000 fd:00 595078 /usr/lib64/libstdc++.so.6.0.8 2ab40ebea000-2ab40ebed000 rw-p 000eb000 fd:00 595078 /usr/lib64/libstdc++.so.6.0.8 2ab40ebed000-2ab40ebff000 rw-p 2ab40ebed000 00:00 0 2ab40ebff000-2ab40ec81000 r-xp 00000000 fd:00 1241866 /lib64/libm-2.5.so 2ab40ec81000-2ab40ee80000 ---p 00082000 fd:00 1241866 /lib64/libm-2.5.so 2ab40ee80000-2ab40ee81000 r--p 00081000 fd:00 1241866 /lib64/libm-2.5.so 2ab40ee81000-2ab40ee82000 rw-p 00082000 fd:00 1241866 /lib64/libm-2.5.so 2ab40ee82000-2ab40ee83000 rw-p 2ab40ee82000 00:00 0 2ab40ee83000-2ab40efd2000 r-xp 00000000 fd:00 1241545 /lib64/libc-2.5.so 2ab40efd2000-2ab40f1d2000 ---p 0014f000 fd:00 1241545 /lib64/libc-2.5.so 2ab40f1d2000-2ab40f1d6000 r--p 0014f000 fd:00 1241545 /lib64/libc-2.5.so 2ab40f1d6000-2ab40f1d7000 rw-p 00153000 fd:00 1241545 /lib64/libc-2.5.so 2ab40f1d7000-2ab40f1dd000 rw-p 2ab40f1d7000 00:00 0 2ab40f1dd000-2ab40f1df000 r-xp 00000000 fd:00 1241722 /lib64/libcom_err.so.2.1 2ab40f1df000-2ab40f3de000 ---p 00002000 fd:00 1241722 /lib64/libcom_err.so.2.1 2ab40f3de000-2ab40f3df000 rw-p 00001000 fd:00 1241722 /lib64/libcom_err.so.2.1 2ab40f3df000-2ab40f3ec000 r-xp 00000000 fd:00 1895684 /lib64/libgcc_s-4.1.2-20080825.so.1 2ab40f3ec000-2ab40f5ec000 ---p 0000d000 fd:00 1895684 /lib64/libgcc_s-4.1.2-20080825.so.1 2ab40f5ec000-2ab40f5ed000 rw-p 0000d000 fd:00 1895684 /lib64/libgcc_s-4.1.2-20080825.so.1 2ab40f5ed000-2ab40f5ee000 rw-p 2ab40f5ed000 00:00 0 2ab40f5ee000-2ab40f5ff000 r-xp 00000000 fd:00 1241871 /lib64/libresolv-2.5.so 2ab40f5ff000-2ab40f7ff000 ---p 00011000 fd:00 1241871 /lib64/libresolv-2.5.so 2ab40f7ff000-2ab40f800000 r--p 00011000 fd:00 1241871 /lib64/libresolv-2.5.so 2ab40f800000-2ab40f801000 rw-p 00012000 fd:00 1241871 /lib64/libresolv-2.5.so 2ab40f801000-2ab40f807000 rw-p 2ab40f801000 00:00 0 2ab40f807000-2ab40f808000 ---p 2ab40f807000 00:00 0 2ab40f808000-2ab410208000 rw-p 2ab40f808000 00:00 0 2ab410208000-2ab41023d000 r--s 00000000 fd:00 784372 /var/db/nscd/passwd 2ab41023d000-2ab410272000 r--s 00000000 fd:00 784374 /var/db/nscd/group 2ab410272000-2ab4393cb000 rw-p 2ab410272000 00:00 0 2ab43c000000-2ab43c021000 rw-p 2ab43c000000 00:00 0 2ab43c021000-2ab440000000 ---p 2ab43c021000 00:00 0 2ab440000000-2ab48376d000 rw-p 2ab440000000 00:00 0 2ab48376d000-2ab48376e000 ---p 2ab48376d000 00:00 0 2ab48376e000-2ab48416e000 rw-p 2ab48376e000 00:00 0 2ab48416e000-2ab48416f000 ---p 2ab48416e000 00:00 0 2ab48416f000-2ab484b6f000 rw-p 2ab48416f000 00:00 0 2ab484b6f000-2ab484b70000 ---p 2ab484b6f000 00:00 0 2ab484b70000-2ab485570000 rw-p 2ab484b70000 00:00 0 2ab485570000-2ab485571000 ---p 2ab485570000 00:00 0 2ab485571000-2ab485f71000 rw-p 2ab485571000 00:00 0 2ab485f71000-2ab485f72000 ---p 2ab485f71000 00:00 0 2ab485f72000-2ab486972000 rw-p 2ab485f72000 00:00 0 2ab486972000-2ab486973000 ---p 2ab486972000 00:00 0 2ab486973000-2ab487373000 rw-p 2ab486973000 00:00 0 2ab487373000-2ab487374000 ---p 2ab487373000 00:00 0 2ab487374000-2ab487d74000 rw-p 2ab487374000 00:00 0 2ab487d74000-2ab487d75000 ---p 2ab487d74000 00:00 0 2ab487d75000-2ab488775000 rw-p 2ab487d75000 00:00 0 2ab488775000-2ab488776000 ---p 2ab488775000 00:00 0 2ab488776000-2ab489176000 rw-p 2ab488776000 00:00 0 2ab489176000-2ab489177000 ---p 2ab489176000 00:00 0 2ab489177000-2ab489b77000 rw-p 2ab489177000 00:00 0 2ab489b77000-2ab489b78000 ---p 2ab489b77000 00:00 0 2ab489b78000-2ab48a578000 rw-p 2ab489b78000 00:00 0 2ab48ac59000-2ab48b299000 rw-p 2ab48ac59000 00:00 0 2ab48b299000-2ab48b29a000 ---p 2ab48b299000 00:00 0 2ab48b29a000-2ab48bc9a000 rw-p 2ab48b29a000 00:00 0 2ab48bc9a000-2ab48bc9b000 ---p 2ab48bc9a000 00:00 0 2ab48bc9b000-2ab48c69b000 rw-p 2ab48bc9b000 00:00 0 2ab48c69b000-2ab48c69c000 ---p 2ab48c69b000 00:00 0 2ab48c69c000-2ab48d09c000 rw-p 2ab48c69c000 00:00 0 2ab48d09c000-2ab48d09d000 ---p 2ab48d09c000 00:00 0 2ab48d09d000-2ab48da9d000 rw-p 2ab48d09d000 00:00 0 2ab48da9d000-2ab48da9e000 ---p 2ab48da9d000 00:00 0 2ab48da9e000-2ab48e49e000 rw-p 2ab48da9e000 00:00 0 2ab48e49e000-2ab48e49f000 ---p 2ab48e49e000 00:00 0 2ab48e49f000-2ab48e4e7000 rw-p 2ab48e49f000 00:00 0 2ab48e4e7000-2ab48e4e8000 ---p 2ab48e4e7000 00:00 0 2ab48e4e8000-2ab48e530000 rw-p 2ab48e4e8000 00:00 0 2ab48e530000-2ab48e531000 ---p 2ab48e530000 00:00 0 2ab48e531000-2ab48e579000 rw-p 2ab48e531000 00:00 0 2ab48e579000-2ab48e57a000 ---p 2ab48e579000 00:00 0 2ab48e57a000-2ab48e5c2000 rw-p 2ab48e57a000 00:00 0 2ab48e5c2000-2ab48e7db000 r-xp 00000000 fd:00 882968 /usr/lib64/galera/libgalera_smm.so 2ab48e7db000-2ab48e9da000 ---p 00219000 fd:00 882968 /usr/lib64/galera/libgalera_smm.so 2ab48e9da000-2ab48e9e5000 rw-p 00218000 fd:00 882968 /usr/lib64/galera/libgalera_smm.so 2ab48e9e5000-2ab48e9e6000 rw-p 2ab48e9e5000 00:00 0 2ab48e9e6000-2ab4ce9e7000 rw-s 00000000 fd:00 1013094 /var/lib/mysql/galera.cache 2ab4ce9e7000-2ab4ce9e8000 ---p 2ab4ce9e7000 00:00 0 2ab4ce9e8000-2ab4cf3e9000 rw-p 2ab4ce9e8000 00:00 0 7fffd4dc9000-7fffd4dde000 rw-p 7ffffffe9000 00:00 0 [stack] 7fffd4de6000-7fffd4de9000 r-xp 7fffd4de6000 00:00 0 [vdso] ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vsyscall] 130228 20:30:48 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 5.5.28a-MariaDB-log key_buffer_size=536870912 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 724320 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x14141f00 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x2ab48e5780a8 thread_stack 0x48000 ??:0(my_print_stacktrace)[0xa9a3ae] ??:0(handle_fatal_signal)[0x6e383b] ??:0(??)[0x2ab40de96ca0] ??:0(??)[0x2ab40eeb32c5] ??:0(??)[0x2ab40eeb4d70] ??:0(??)[0x2ab40eeecb4b] ??:0(??)[0x2ab40eef46af] ??:0(??)[0x2ab40eef88db] ??:0(??)[0x2ab48e76d968] ??:0(wsrep_free_status(THD*))[0x6993b8] ??:0(mysql_execute_command(THD*))[0x595629] ??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x596f39] ??:0(_ZL17wsrep_mysql_parseP3THDPcjP12Parser_state)[0x59780e] ??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x599d83] ??:0(do_command(THD*))[0x59a309] ??:0(do_handle_one_connection(THD*))[0x6505a3] ??:0(handle_one_connection)[0x65075c] ??:0(??)[0x2ab40de8e83d] ??:0(??)[0x2ab40ef5803d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x1414e1b8): SHOW STATUS LIKE 'wsrep_%' Connection ID (thread ID): 2 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 130228 20:30:48 mysqld_safe Number of processes running now: 0