Contrail 3.0.2-51: mysql db crash in one of the control nodes

[Deepak Jeyaraman]

Version used:

root@ccra-17:~/contrail_bug_1627369# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.0.2.0-51 51
contrail-config 3.0.2.0-51 51
contrail-config-openstack 3.0.2.0-51 51
contrail-control 3.0.2.0-51 51
contrail-dns 3.0.2.0-51 51
contrail-docs 3.0.2.0-51 51

Setup:

3 contrail config/control/analytics node , 15 compute nodes

Issue:

mysql db has crashed in one of the nodes all of a sudden.

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@161103 12:51:59 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
161103 12:51:59 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.TNSqqxrZ1s
161103 12:52:04 mysqld_safe WSREP: Recovered position 9790a0fd-9710-11e6-b6f8-aa5f81d3d7ac:15572118
161103 12:52:04 [Note] WSREP: wsrep_start_position var submitted: '9790a0fd-9710-11e6-b6f8-aa5f81d3d7ac:15572118'
161103 12:52:04 [Note] WSREP: Read nil XID from storage engines, skipping position init
161103 12:52:04 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
161103 12:52:04 [Note] WSREP: wsrep_load(): Galera 25.2.9(r172) by Codership Oy <email address hidden> loaded successfully.
161103 12:52:04 [Note] WSREP: Found saved state: 9790a0fd-9710-11e6-b6f8-aa5f81d3d7ac:8750028
161103 12:52:04 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.

161103 12:52:05 [Note] WSREP: declaring 9d6a4aa1-9710-11e6-a9fa-fa94e9a90c62 stable
161103 12:52:05 [Note] WSREP: declaring ce1ce921-9714-11e6-8543-5bceb49b527a stable
161103 12:52:05 [Note] WSREP: Node 9d6a4aa1-9710-11e6-a9fa-fa94e9a90c62 state prim
161103 12:52:05 [Note] WSREP: view(view_id(PRIM,9d6a4aa1-9710-11e6-a9fa-fa94e9a90c62,17) memb {
        9d6a4aa1-9710-11e6-a9fa-fa94e9a90c62,
        ce1ce921-9714-11e6-8543-5bceb49b527a,
        fe688299-a1fe-11e6-be5e-ef435516252a,
} joined {
} left {
} partitioned {
})

161103 12:52:18 [Note] WSREP: Receiving IST: 6822985 writesets, seqnos 8750028-15573013
161103 12:52:18 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.37' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL), wsrep_25.10.r3990
161103 12:52:29 [Note] /usr/sbin/mysqld: Normal shutdown

161103 12:52:29 [Note] WSREP: Stop replication
161103 12:52:29 [Note] WSREP: Closing send monitor...
161103 12:52:29 [Note] WSREP: Closed send monitor.
161103 12:52:29 [Note] WSREP: gcomm: terminating thread
161103 12:52:29 [Note] WSREP: gcomm: joining thread
161103 12:52:29 [Note] WSREP: gcomm: closing backend
161103 12:52:29 [Note] WSREP: view(view_id(NON_PRIM,9d6a4aa1-9710-11e6-a9fa-fa94e9a90c62,17) memb {
        fe688299-a1fe-11e6-be5e-ef435516252a,
} joined {
} left {
} partitioned {
        9d6a4aa1-9710-11e6-a9fa-fa94e9a90c62,
        ce1ce921-9714-11e6-8543-5bceb49b527a,
})

Logs are stored here:

10.102.29.129

root@ccra-17:~/PR-1639308# ls -ltr
total 1144748
-rw-r----- 1 root root 1170336220 Nov 4 10:24 ccra-17-error.log
-rw-r----- 1 root root 1265650 Nov 4 10:24 ccra-16-error.log
-rw-r----- 1 root root 613341 Nov 4 10:24 ccra-14-error.log

[Ranjeet R (rranjeet-n)]

MySQL crashes are not in our purview of control. We have to manually bring up MySQL if CMON fails to bring it up.