Master crashed when slave IST failed

[Vadim Tkachenko]

This bug led to https://bugs.launchpad.net/codership-mysql/+bug/908021

The setup is node A in EC2 west-US, the node B is in EC2 west-EU.

Due to misconfiguration the nodeB has no proper IST config.

NodeA config:

[mysqld]
datadir=/mnt/data
user=mysql

binlog_format=ROW

wsrep_provider=/usr/lib64/libgalera_smm.so

wsrep_cluster_address=gcomm://

wsrep_slave_threads=2
wsrep_cluster_name=trimethylxanthine
wsrep_sst_method=rsync
wsrep_node_name=node1

innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2

NodeB config:

[mysqld]
datadir=/mnt/data
user=mysql

binlog_format=ROW

wsrep_provider=/usr/lib64/libgalera_smm.so

wsrep_cluster_address=gcomm://ec2-50-18-38-99.us-west-1.compute.amazonaws.com
wsrep_sst_receive_address=ec2-176-34-205-113.eu-west-1.compute.amazonaws.com:4020

wsrep_slave_threads=2
wsrep_cluster_name=trimethylxanthine
wsrep_sst_method=rsync
wsrep_node_name=node2

innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2

Now when we start NodeB and it tries to do IST, I have a crashed NodeA.

0x00007fb9c7e80905 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x00007fb9c7e80905 in raise () from /lib64/libc.so.6
#1 0x00007fb9c7e820e5 in abort () from /lib64/libc.so.6
#2 0x00007fb9c62c7a7d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3 0x00007fb9c62c5c06 in ?? () from /usr/lib64/libstdc++.so.6
#4 0x00007fb9c62c5c33 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5 0x00007fb9c62c5c46 in ?? () from /usr/lib64/libstdc++.so.6
#6 0x00007fb9c62c52d3 in __cxa_call_unexpected () from /usr/lib64/libstdc++.so.6
#7 0x00007fb9c6b607e3 in galera::ReplicatorSMM::process_state_req (this=0x302d500, recv_ctx=0x7fb9a0000990, req=0x7fb9b5bf8ec0,
    req_size=140435594907248, seqno_l=0, donor_seq=3) at galera/src/replicator_str.cpp:254
#8 0x00007fb9c6b35f1c in galera::GcsActionSource::dispatch (this=0x302da00, recv_ctx=0x7fb9a0000990, act=...)
    at galera/src/gcs_action_source.cpp:138
#9 0x00007fb9c6b366e8 in galera::GcsActionSource::process (this=0x302da00, recv_ctx=0x7fb9a0000990)
    at galera/src/gcs_action_source.cpp:166
#10 0x00007fb9c6b533ad in galera::ReplicatorSMM::async_recv (this=0x302d500, recv_ctx=0x7fb9a0000990)
    at galera/src/replicator_smm.cpp:350
#11 0x00007fb9c6b67893 in galera_recv (gh=<value optimized out>, recv_ctx=<value optimized out>)
    at galera/src/wsrep_provider.cpp:196
#12 0x000000000058c9eb in wsrep_replication_process(THD*) ()
#13 0x000000000051abc3 in start_wsrep_THD ()
#14 0x00007fb9c8cac7e1 in start_thread () from /lib64/libpthread.so.0
#15 0x00007fb9c7f3377d in clone () from /lib64/libc.so.6

Log from NodeA

111223 1:58:34 [Note] WSREP: GMCast::handle_stable_view: view(view_id(PRIM,872108f6-2d33-11e1-0800-85cd5691abbc,4) memb {
        872108f6-2d33-11e1-0800-85cd5691abbc,
        b5d57d4e-2d32-11e1-0800-02fc377ed738,
} joined {
} left {
} partitioned {
})
111223 1:58:34 [Note] WSREP: New COMPONENT: primary = yes, my_idx = 1, memb_num = 2
111223 1:58:34 [Note] WSREP: declaring 872108f6-2d33-11e1-0800-85cd5691abbc stable
111223 1:58:34 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
111223 1:58:35 [Note] WSREP: STATE EXCHANGE: sent state msg: 8806c568-2d33-11e1-0800-64efb3309d2b
111223 1:58:35 [Note] WSREP: STATE EXCHANGE: got state msg: 8806c568-2d33-11e1-0800-64efb3309d2b from 0 (node2)
111223 1:58:35 [Note] WSREP: STATE EXCHANGE: got state msg: 8806c568-2d33-11e1-0800-64efb3309d2b from 1 (node1)
111223 1:58:35 [Note] WSREP: Quorum results:
        version = 2,
        component = PRIMARY,
        conf_id = 3,
        members = 1/2 (joined/total),
        act_id = 3,
        last_appl. = 0,
        protocols = 0/2/1 (gcs/repl/appl),
        group UUID = b5d62c24-2d32-11e1-0800-c13d5713abc7
111223 1:58:35 [Note] WSREP: Flow-control interval: [12, 23]
111223 1:58:35 [Note] WSREP: New cluster view: global state: b5d62c24-2d32-11e1-0800-c13d5713abc7:3, view# 4: Primary, number of nodes: 2, my index: 1, protocol version 1
111223 1:58:35 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
111223 1:58:35 [Note] WSREP: Assign initial position for certification: 3, protocol version: 1
111223 1:58:37 [Note] WSREP: Node 0 (node2) requested state transfer from '*any*'. Selected 1 (node1)(SYNCED) as donor.
111223 1:58:37 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 3)
111223 1:58:37 [Note] WSREP: IST request: b5d62c24-2d32-11e1-0800-c13d5713abc7:1-3|tcp://10.237.75.233:4568
111223 1:58:37 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
111223 1:58:37 [Note] WSREP: Running: 'wsrep_sst_rsync 'donor' 'ec2-176-34-205-113.eu-west-1.compute.amazonaws.com:4020/rsync_sst' '(null)' '/mnt/data/' '/etc/my.cnf' 'b5d62c24-2d32-11e1-0800-c13d5713abc7' '1' '1' 2>sst.err'
111223 1:58:37 [Note] WSREP: sst_donor_thread signaled with 0
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<asio::system_error> >'
  what(): Network is unreachable
111223 1:59:36 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=4
max_threads=151
thread_count=3
connection_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338648 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fb9a0000990
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fb9b5bf9e78 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x7b7c79]
/usr/sbin/mysqld(handle_segfault+0x464)[0x5140c4]
/lib64/libpthread.so.0(+0xf490)[0x7fb9c8cb4490]
/lib64/libc.so.6(gsignal+0x35)[0x7fb9c7e80905]
/lib64/libc.so.6(abort+0x175)[0x7fb9c7e820e5]
111223 1:59:36 [Warning] WSREP: last inactive check more than PT1.5S ago, skipping check
/usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x12d)[0x7fb9c62c7a7d]
/usr/lib64/libstdc++.so.6(+0xbcc06)[0x7fb9c62c5c06]
/usr/lib64/libstdc++.so.6(+0xbcc33)[0x7fb9c62c5c33]
/usr/lib64/libstdc++.so.6(+0xbcc46)[0x7fb9c62c5c46]
/usr/lib64/libstdc++.so.6(__cxa_call_unexpected+0x43)[0x7fb9c62c52d3]
/usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM17process_state_reqEPvPKvmll+0x1073)[0x7fb9c6b607e3]
/usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_action+0x3dc)[0x7fb9c6b35f1c]
/usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPv+0x58)[0x7fb9c6b366e8]
/usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0xfd)[0x7fb9c6b533ad]
/usr/lib64/libgalera_smm.so(galera_recv+0x23)[0x7fb9c6b67893]
/usr/sbin/mysqld(_Z25wsrep_replication_processP3THD+0x6b)[0x58c9eb]
/usr/sbin/mysqld(start_wsrep_THD+0x3f3)[0x51abc3]
/lib64/libpthread.so.0(+0x77e1)[0x7fb9c8cac7e1]
/lib64/libc.so.6(clone+0x6d)[0x7fb9c7f3377d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query ((nil)): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Log from NodeB

111223 1:58:33 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/libgalera_smm.so'
111223 1:58:33 [Note] WSREP: wsrep_load(): Galera 2.0beta(r99) by Codership Oy <email address hidden> loaded succesfully.
111223 1:58:33 [Note] WSREP: Reusing existing '/mnt/data//galera.cache'.
111223 1:58:33 [Note] WSREP: Passing config to GCS: gcache.dir = /mnt/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /mnt/data//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; replicator.commit_order = 3
111223 1:58:33 [Note] WSREP: wsrep_sst_grab()
111223 1:58:33 [Note] WSREP: Start replication
111223 1:58:33 [Note] WSREP: Found saved state: b5d62c24-2d32-11e1-0800-c13d5713abc7:1
111223 1:58:33 [Note] WSREP: Assign initial position for certification: 1, protocol version: -1
111223 1:58:33 [Note] WSREP: Setting initial position to b5d62c24-2d32-11e1-0800-c13d5713abc7:1
111223 1:58:33 [Note] WSREP: protonet asio version 0
111223 1:58:33 [Note] WSREP: backend: asio
111223 1:58:33 [Note] WSREP: GMCast version 0
111223 1:58:33 [Note] WSREP: (872108f6-2d33-11e1-0800-85cd5691abbc, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
111223 1:58:33 [Note] WSREP: (872108f6-2d33-11e1-0800-85cd5691abbc, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
111223 1:58:33 [Note] WSREP: EVS version 0
111223 1:58:33 [Note] WSREP: PC version 0
111223 1:58:33 [Note] WSREP: gcomm: connecting to group 'trimethylxanthine', peer 'ec2-50-18-38-99.us-west-1.compute.amazonaws.com:'
111223 1:58:34 [Note] WSREP: GMCast::handle_stable_view: view(view_id(PRIM,872108f6-2d33-11e1-0800-85cd5691abbc,4) memb {
        872108f6-2d33-11e1-0800-85cd5691abbc,
        b5d57d4e-2d32-11e1-0800-02fc377ed738,
} joined {
} left {
} partitioned {
})
111223 1:58:34 [Note] WSREP: declaring b5d57d4e-2d32-11e1-0800-02fc377ed738 stable
111223 1:58:34 [Note] WSREP: gcomm: connected
111223 1:58:34 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
111223 1:58:34 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
111223 1:58:34 [Note] WSREP: Opened channel 'trimethylxanthine'
111223 1:58:34 [Note] WSREP: New COMPONENT: primary = yes, my_idx = 0, memb_num = 2
111223 1:58:34 [Note] WSREP: Waiting for SST to complete.
111223 1:58:34 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 8806c568-2d33-11e1-0800-64efb3309d2b
111223 1:58:34 [Note] WSREP: STATE EXCHANGE: sent state msg: 8806c568-2d33-11e1-0800-64efb3309d2b
111223 1:58:34 [Note] WSREP: STATE EXCHANGE: got state msg: 8806c568-2d33-11e1-0800-64efb3309d2b from 0 (node2)
111223 1:58:34 [Note] WSREP: STATE EXCHANGE: got state msg: 8806c568-2d33-11e1-0800-64efb3309d2b from 1 (node1)
111223 1:58:34 [Note] WSREP: Quorum results:
        version = 2,
        component = PRIMARY,
        conf_id = 3,
        members = 1/2 (joined/total),
        act_id = 3,
        last_appl. = -1,
        protocols = 0/2/1 (gcs/repl/appl),
        group UUID = b5d62c24-2d32-11e1-0800-c13d5713abc7
111223 1:58:34 [Note] WSREP: Flow-control interval: [12, 23]
111223 1:58:34 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 3)
111223 1:58:34 [Note] WSREP: New cluster view: global state: b5d62c24-2d32-11e1-0800-c13d5713abc7:3, view# 4: Primary, number of nodes: 2, my index: 0, protocol version 1
111223 1:58:34 [Warning] WSREP: Gap in state sequence. Need state transfer.
111223 1:58:37 [Note] WSREP: Running: 'wsrep_sst_rsync 'joiner' 'ec2-176-34-205-113.eu-west-1.compute.amazonaws.com:4020' '' '/mnt/data/' '/etc/my.cnf' '1929' 2>sst.err'
111223 1:58:37 [Note] WSREP: Prepared SST request: rsync|ec2-176-34-205-113.eu-west-1.compute.amazonaws.com:4020/rsync_sst
111223 1:58:37 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
111223 1:58:37 [Note] WSREP: Assign initial position for certification: 3, protocol version: 1
111223 1:58:37 [Note] WSREP: prepared IST receiver, listening in: tcp://10.237.75.233:4568
111223 1:58:37 [Note] WSREP: Node 0 (node2) requested state transfer from '*any*'. Selected 1 (node1)(SYNCED) as donor.
111223 1:58:37 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 3)
111223 1:58:37 [Note] WSREP: Requesting state transfer: success, donor: 1
111223 1:58:39 [Note] WSREP: SST complete, seqno: 1
111223 1:58:39 [Note] Plugin 'FEDERATED' is disabled.
111223 1:58:39 InnoDB: The InnoDB memory heap is disabled
111223 1:58:39 InnoDB: Mutexes and rw_locks use GCC atomic builtins
111223 1:58:39 InnoDB: Compressed tables use zlib 1.2.3
111223 1:58:39 InnoDB: Using Linux native AIO
111223 1:58:39 InnoDB: Initializing buffer pool, size = 128.0M
111223 1:58:39 InnoDB: Completed initialization of buffer pool
111223 1:58:39 InnoDB: highest supported file format is Barracuda.
111223 1:58:39 InnoDB: Waiting for the background threads to start
111223 1:58:40 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 1605728
111223 1:58:40 [Note] Event Scheduler: Loaded 0 events
111223 1:58:40 [Note] WSREP: Signalling provider to continue.
111223 1:58:40 [Note] WSREP: Received SST: b5d62c24-2d32-11e1-0800-c13d5713abc7:1
111223 1:58:40 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.17' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona XtraDB Cluster (GPL), Release alpha22.1, Revision 3673 wsrep_22.3.r3673
111223 1:58:40 [Note] WSREP: SST finished: b5d62c24-2d32-11e1-0800-c13d5713abc7:1
111223 1:58:40 [Note] WSREP: Receiving IST: 2 writesets, seqnos 1-3
111223 1:58:40 [Note] WSREP: (872108f6-2d33-11e1-0800-85cd5691abbc, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://50.18.38.99:4567
111223 1:58:41 [Note] WSREP: (872108f6-2d33-11e1-0800-85cd5691abbc, 'tcp://0.0.0.0:4567') reconnecting to b5d57d4e-2d32-11e1-0800-02fc377ed738 (tcp://50.18.38.99:4567), attempt 0
111223 1:58:42 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, OPERATIONAL, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:43 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:43 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:44 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:44 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:45 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:45 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:46 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:46 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:47 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:47 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:48 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:48 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:49 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:49 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:50 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:50 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:51 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:51 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:52 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) suspecting node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:52 [Note] WSREP: evs::proto(872108f6-2d33-11e1-0800-85cd5691abbc, GATHER, view_id(REG,872108f6-2d33-11e1-0800-85cd5691abbc,4)) detected inactive node: b5d57d4e-2d32-11e1-0800-02fc377ed738
111223 1:58:52 [Note] WSREP: New COMPONENT: primary = no, my_idx = 0, memb_num = 1
111223 1:58:52 [Note] WSREP: Flow-control interval: [8, 16]
111223 1:58:52 [Note] WSREP: Received NON-PRIMARY.
111223 1:58:52 [Note] WSREP: Shifting JOINER -> OPEN (TO: 3)
111223 1:58:52 [Note] WSREP: GMCast::handle_stable_view: view(view_id(NON_PRIM,872108f6-2d33-11e1-0800-85cd5691abbc,4) memb {
        872108f6-2d33-11e1-0800-85cd5691abbc,
} joined {
} left {
} partitioned {
        b5d57d4e-2d32-11e1-0800-02fc377ed738,
})
111223 1:58:52 [Note] WSREP: New COMPONENT: primary = no, my_idx = 0, memb_num = 1
111223 1:58:52 [Note] WSREP: Flow-control interval: [8, 16]
111223 1:58:52 [Note] WSREP: Received NON-PRIMARY.
111223 1:58:52 [Note] WSREP: GMCast::handle_stable_view: view(view_id(NON_PRIM,872108f6-2d33-11e1-0800-85cd5691abbc,5) memb {
        872108f6-2d33-11e1-0800-85cd5691abbc,
} joined {
} left {
} partitioned {
        b5d57d4e-2d32-11e1-0800-02fc377ed738,
})
111223 2:00:01 [Note] WSREP: (872108f6-2d33-11e1-0800-85cd5691abbc, 'tcp://0.0.0.0:4567') reconnecting to b5d57d4e-2d32-11e1-0800-02fc377ed738 (tcp://50.18.38.99:4567), attempt 30

[Teemu Ollakka (teemu-ollakka)]

Fix pushed in lp:galera/2.x revision 107