kolla

Bug #1672240
Comment #11

Comment 11 for bug 1672240

Revision history for this message

MarginHu (margin2017) wrote on 2017-03-17:

#11

[root@kode1 mariadb]# tail -n 100 mariadb.log
        0745c975,0
        1c699739,0
        319270e6,0
        46b83db9,0
        5bdeb2c7,0
        5dc7f26e,0
        7338683d,0
        885fabba,0
        9d856ffe,0
        9eed86eb,0
        b2ae1580,0
        c7cff5ac,0
        dcf41ff9,0
        f21f343e,0
})
170317 21:48:49 [Note] WSREP: (710c31d9, 'tcp://192.168.102.21:4567') turning message relay requesting off
170317 21:49:16 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
         at gcomm/src/pc.cpp:connect():158
170317 21:49:16 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
170317 21:49:16 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1380: Failed to open channel 'openstack' at 'gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567': -110 (Connection timed out)
170317 21:49:16 [ERROR] WSREP: gcs connect failed: Connection timed out
170317 21:49:16 [ERROR] WSREP: wsrep::connect(gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567) failed: 7
170317 21:49:16 [ERROR] Aborting

170317 21:49:16 [Note] WSREP: Service disconnected.
170317 21:49:17 [Note] WSREP: Some threads may fail to exit.
170317 21:49:17 [Note] /usr/sbin/mysqld: Shutdown complete

170317 21:49:17 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended
170317 21:49:18 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
170317 21:49:18 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql//wsrep_recovery.c0BcOk' --pid-file='/var/lib/mysql//kode1.genomics.cn-recover.pid'
nohup: ignoring input
170317 21:49:18 [Note] /usr/sbin/mysqld (mysqld 10.0.29-MariaDB-wsrep) starting as process 179 ...
170317 21:49:21 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
170317 21:49:21 [Note] WSREP: Read nil XID from storage engines, skipping position init
170317 21:49:21 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
170317 21:49:21 [Note] /usr/sbin/mysqld (mysqld 10.0.29-MariaDB-wsrep) starting as process 217 ...
170317 21:49:21 [Note] WSREP: wsrep_load(): Galera 25.3.19(r3667) by Codership Oy <email address hidden> loaded successfully.
170317 21:49:21 [Note] WSREP: CRC-32C: using hardware acceleration.
170317 21:49:21 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootsrap: 1
170317 21:49:21 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.102.21; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://192.168.102.21:4567; gmcast.segment = 0; gmcast.version = 0; ist
170317 21:49:21 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(00000000-0000-0000-0000-000000000000:-1)
170317 21:49:21 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
170317 21:49:21 [Note] WSREP: wsrep_sst_grab()
170317 21:49:21 [Note] WSREP: Start replication
170317 21:49:21 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
170317 21:49:21 [Note] WSREP: protonet asio version 0
170317 21:49:21 [Note] WSREP: Using CRC-32C for message checksums.
170317 21:49:21 [Note] WSREP: backend: asio
170317 21:49:21 [Note] WSREP: gcomm thread scheduling priority set to other:0
170317 21:49:21 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
170317 21:49:21 [Note] WSREP: restore pc from disk failed
170317 21:49:21 [Note] WSREP: GMCast version 0
170317 21:49:21 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') listening at tcp://192.168.102.21:4567
170317 21:49:21 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') multicast: , ttl: 1
170317 21:49:21 [Note] WSREP: EVS version 0
170317 21:49:21 [Note] WSREP: gcomm: connecting to group 'openstack', peer '192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567'
170317 21:49:21 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') connection established to 9fd9d50c tcp://192.168.102.22:4567
170317 21:49:21 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') turning message relay requesting on, nonlive peers:
170317 21:49:21 [Note] WSREP: declaring 9fd9d50c at tcp://192.168.102.22:4567 stable
170317 21:49:21 [Note] WSREP: view(view_id(NON_PRIM,863108cf,40) memb {
        863108cf,0
        9fd9d50c,0
} joined {
} left {
} partitioned {
        0745c975,0
        1c699739,0
        319270e6,0
        46b83db9,0
        5bdeb2c7,0
        5dc7f26e,0
        710c31d9,0
        7338683d,0
        885fabba,0
        9d856ffe,0
        9eed86eb,0
        b2ae1580,0
        c7cff5ac,0
        dcf41ff9,0
        f21f343e,0
})
170317 21:49:25 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') turning message relay requesting off
170317 21:49:52 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
         at gcomm/src/pc.cpp:connect():158
170317 21:49:52 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
170317 21:49:52 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1380: Failed to open channel 'openstack' at 'gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567': -110 (Connection timed out)
170317 21:49:52 [ERROR] WSREP: gcs connect failed: Connection timed out
170317 21:49:52 [ERROR] WSREP: wsrep::connect(gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567) failed: 7
170317 21:49:52 [ERROR] Aborting

170317 21:49:52 [Note] WSREP: Service disconnected.
170317 21:49:53 [Note] WSREP: Some threads may fail to exit.
170317 21:49:53 [Note] /usr/sbin/mysqld: Shutdown complete

170317 21:49:53 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended
170317 21:49:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
170317 21:49:53 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql//wsrep_recovery.xhDg6X' --pid-file='/var/lib/mysql//kode1.genomics.cn-recover.pid'
nohup: ignoring input
170317 21:49:53 [Note] /usr/sbin/mysqld (mysqld 10.0.29-MariaDB-wsrep) starting as process 178 ...

[root@kode1 docker]# docker ps 
CONTAINER ID        IMAGE                                                         COMMAND             CREATED             STATUS                         PORTS               NAMES
9df398bdc781        192.168.103.16:5000/bgi/centos-binary-mariadb:4.0.0.1         "kolla_start"       2 minutes ago       Restarting (0) 4 seconds ago                       mariadb
bf5d97a8776a        192.168.103.16:5000/bgi/centos-binary-memcached:4.0.0.1       "kolla_start"       2 minutes ago       Up 2 minutes                                       memcached
0ea71f1067da        192.168.103.16:5000/bgi/centos-binary-kibana:4.0.0.1          "kolla_start"       2 minutes ago       Up 2 minutes                                       kibana
b7ed891f11eb        192.168.103.16:5000/bgi/centos-binary-elasticsearch:4.0.0.1   "kolla_start"       4 minutes ago       Up 4 minutes                                       elasticsearch
578467f2157e        192.168.103.16:5000/bgi/centos-binary-cron:4.0.0.1            "kolla_start"       4 minutes ago       Up 4 minutes                                       cron
d48b5b4ab9d5        192.168.103.16:5000/bgi/centos-binary-kolla-toolbox:4.0.0.1   "kolla_start"       4 minutes ago       Up 4 minutes                                       kolla_toolbox
1c6ecbcb451c        192.168.103.16:5000/bgi/centos-binary-fluentd:4.0.0.1         "kolla_start"       4 minutes ago       Up 4 minutes                                       fluentd
[root@kode1 docker]# cd /var/lib/docker/volumes/kolla_logs/_data/mariadb/
[root@kode1 mariadb]# ls
mariadb.log
 
[root@kode1 mariadb]# tail -n 100 mariadb.log
        0745c975,0
        1c699739,0
        319270e6,0
        46b83db9,0
        5bdeb2c7,0
        5dc7f26e,0
        7338683d,0
        885fabba,0
        9d856ffe,0
        9eed86eb,0
        b2ae1580,0
        c7cff5ac,0
        dcf41ff9,0
        f21f343e,0
})
170317 21:48:49 [Note] WSREP: (710c31d9, 'tcp://192.168.102.21:4567') turning message relay requesting off
170317 21:49:16 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
         at gcomm/src/pc.cpp:connect():158
170317 21:49:16 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
170317 21:49:16 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1380: Failed to open channel 'openstack' at 'gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567': -110 (Connection timed out)
170317 21:49:16 [ERROR] WSREP: gcs connect failed: Connection timed out
170317 21:49:16 [ERROR] WSREP: wsrep::connect(gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567) failed: 7
170317 21:49:16 [ERROR] Aborting

170317 21:49:16 [Note] WSREP: Service disconnected.
170317 21:49:17 [Note] WSREP: Some threads may fail to exit.
170317 21:49:17 [Note] /usr/sbin/mysqld: Shutdown complete

170317 21:49:17 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended
170317 21:49:18 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
170317 21:49:18 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql//wsrep_recovery.c0BcOk' --pid-file='/var/lib/mysql//kode1.genomics.cn-recover.pid'
nohup: ignoring input
170317 21:49:18 [Note] /usr/sbin/mysqld (mysqld 10.0.29-MariaDB-wsrep) starting as process 179 ...
170317 21:49:21 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
170317 21:49:21 [Note] WSREP: Read nil XID from storage engines, skipping position init
170317 21:49:21 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
170317 21:49:21 [Note] /usr/sbin/mysqld (mysqld 10.0.29-MariaDB-wsrep) starting as process 217 ...
170317 21:49:21 [Note] WSREP: wsrep_load(): Galera 25.3.19(r3667) by Codership Oy <info@codership.com> loaded successfully.
170317 21:49:21 [Note] WSREP: CRC-32C: using hardware acceleration.
170317 21:49:21 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootsrap: 1
170317 21:49:21 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.102.21; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://192.168.102.21:4567; gmcast.segment = 0; gmcast.version = 0; ist
170317 21:49:21 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(00000000-0000-0000-0000-000000000000:-1)
170317 21:49:21 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
170317 21:49:21 [Note] WSREP: wsrep_sst_grab()
170317 21:49:21 [Note] WSREP: Start replication
170317 21:49:21 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
170317 21:49:21 [Note] WSREP: protonet asio version 0
170317 21:49:21 [Note] WSREP: Using CRC-32C for message checksums.
170317 21:49:21 [Note] WSREP: backend: asio
170317 21:49:21 [Note] WSREP: gcomm thread scheduling priority set to other:0 
170317 21:49:21 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
170317 21:49:21 [Note] WSREP: restore pc from disk failed
170317 21:49:21 [Note] WSREP: GMCast version 0
170317 21:49:21 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') listening at tcp://192.168.102.21:4567
170317 21:49:21 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') multicast: , ttl: 1
170317 21:49:21 [Note] WSREP: EVS version 0
170317 21:49:21 [Note] WSREP: gcomm: connecting to group 'openstack', peer '192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567'
170317 21:49:21 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') connection established to 9fd9d50c tcp://192.168.102.22:4567
170317 21:49:21 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') turning message relay requesting on, nonlive peers: 
170317 21:49:21 [Note] WSREP: declaring 9fd9d50c at tcp://192.168.102.22:4567 stable
170317 21:49:21 [Note] WSREP: view(view_id(NON_PRIM,863108cf,40) memb {
        863108cf,0
        9fd9d50c,0
} joined {
} left {
} partitioned {
        0745c975,0
        1c699739,0
        319270e6,0
        46b83db9,0
        5bdeb2c7,0
        5dc7f26e,0
        710c31d9,0
        7338683d,0
        885fabba,0
        9d856ffe,0
        9eed86eb,0
        b2ae1580,0
        c7cff5ac,0
        dcf41ff9,0
        f21f343e,0
})
170317 21:49:25 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') turning message relay requesting off
170317 21:49:52 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
         at gcomm/src/pc.cpp:connect():158
170317 21:49:52 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
170317 21:49:52 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1380: Failed to open channel 'openstack' at 'gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567': -110 (Connection timed out)
170317 21:49:52 [ERROR] WSREP: gcs connect failed: Connection timed out
170317 21:49:52 [ERROR] WSREP: wsrep::connect(gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567) failed: 7
170317 21:49:52 [ERROR] Aborting

170317 21:49:52 [Note] WSREP: Service disconnected.
170317 21:49:53 [Note] WSREP: Some threads may fail to exit.
170317 21:49:53 [Note] /usr/sbin/mysqld: Shutdown complete