Comment 2 for bug 1886003

Revision history for this message
zhipeng liu (zhipengs) wrote :

From log, the second mariadb node could not join the cluster
2020-07-02T11:48:41.232843694Z stderr F 2020-07-02 11:48:41,231 - OpenStack-Helm Mariadb - INFO - b'2020-07-02 11:48:41 140031810905856 [Note] WSREP: Service thread queue flushed.'
2020-07-02T11:48:41.232850901Z stderr F 2020-07-02 11:48:41,231 - OpenStack-Helm Mariadb - INFO - b'2020-07-02 11:48:41 140031772694272 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (85c6ecb7-bc59-11ea-bc4b-d7ca09aff26c): 1 (Operation not permitted)'
2020-07-02T11:48:41.233984948Z stderr F 2020-07-02 11:48:41,233 - OpenStack-Helm Mariadb - INFO - b'\t at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.'
2020-07-02T11:48:41.234002331Z stderr F 2020-07-02 11:48:41,233 - OpenStack-Helm Mariadb - INFO - b"2020-07-02 11:48:41 140031561815808 [Note] WSREP: Member 1.0 (mariadb-server-1.mariadb-discovery.openstack.svc.cluster.local) requested state transfer from '*any*'. Selected 0.0 (mariadb-server-0.mariadb-discovery.openstack.svc.cluster.local)(SYNCED) as donor."
2020-07-02T11:48:41.711004558Z stderr F 2020-07-02 11:48:41,710 - OpenStack-Helm Mariadb - INFO - b'2020-07-02 11:48:41 140031561815808 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)'
2020-07-02T11:48:41.71172359Z stderr F 2020-07-02 11:48:41,710 - OpenStack-Helm Mariadb - INFO - b'2020-07-02 11:48:41 140031772694272 [Note] WSREP: Requesting state transfer: success, donor: 0'
2020-07-02T11:48:41.711946836Z stderr F 2020-07-02 11:48:41,711 - OpenStack-Helm Mariadb - INFO - b'2020-07-02 11:48:41 140031772694272 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 85c6ecb7-bc59-11ea-bc4b-d7ca09aff26c:0'
2020-07-02T11:48:41.71223561Z stderr F 2020-07-02 11:48:41,712 - OpenStack-Helm Mariadb - INFO - b'WSREP_SST: [INFO] WARNING: Stale temporary SST directory: /var/lib/mysql//.sst from previous state transfer. Removing (20200702 11:48:41.707)'
2020-07-02T11:48:41.727112835Z stderr F 2020-07-02 11:48:41,726 - OpenStack-Helm Mariadb - INFO - b'WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20200702 11:48:41.722)'
2020-07-02T11:48:41.731457828Z stderr F 2020-07-02 11:48:41,731 - OpenStack-Helm Mariadb - INFO - b'WSREP_SST: [INFO] Proceeding with SST (20200702 11:48:41.720)'
2020-07-02T11:48:41.731603336Z stderr F 2020-07-02 11:48:41,731 - OpenStack-Helm Mariadb - INFO - b'WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20200702 11:48:41.730)'
2020-07-02T11:48:41.767510019Z stderr F 2020-07-02 11:48:41,767 - OpenStack-Helm Mariadb - INFO - b'WSREP_SST: [INFO] Waiting for SST streaming to complete! (20200702 11:48:41.764)'
2020-07-02T11:48:49.549740892Z stderr F 2020-07-02 11:48:49,549 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap
2020-07-02T11:48:51.75744306Z stderr F 2020-07-02 11:48:51,756 - OpenStack-Helm Mariadb - INFO - b'2020-07-02 11:48:51 140031561815808 [Warning] WSREP: 0.0 (mariadb-server-0.mariadb-discovery.openstack.svc.cluster.local): State transfer to 1.0 (mariadb-server-1.mariadb-discovery.openstack.svc.cluster.local) failed: -22 (Invalid argument)'
2020-07-02T11:48:51.757510396Z stderr F 2020-07-02 11:48:51,757 - OpenStack-Helm Mariadb - INFO - b'2020-07-02 11:48:51 140031561815808 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():775: Will never receive state. Need to abort.'
2020-07-02T11:48:51.757554424Z stderr F 2020-07-02 11:48:51,757 - OpenStack-Helm Mariadb - INFO - b'2020-07-02 11:48:51 140031561815808 [Note] WSREP: gcomm: terminating thread'
2020-07-02T11:48:51.757564238Z stderr F 2020-07-02 11:48:51,757 - OpenStack-Helm Mariadb - INFO - b'2020-07-02 11:48:51 140031561815808 [Note] WSREP: gcomm: joining thread'
2020-07-02T11:48:51.757594318Z stderr F 2020-07-02 11:48:51,757 - OpenStack-Helm Mariadb - INFO - b'2020-07-02 11:48:51 140031561815808 [Note] WSREP: gcomm: closing backend'
2020-07-02T11:48:51.81367173Z stderr F 2020-07-02 11:48:51,813 - OpenStack-Helm Mariadb - INFO - b'WSREP_SST: [ERROR] xtrabackup_checkpoints missing, failed innobackupex/SST on donor (20200702 11:48:51.811)'
2020-07-02T11:48:51.817553673Z stderr F 2020-07-02 11:48:51,817 - OpenStack-Helm Mariadb - INFO - b'WSREP_SST: [ERROR] Cleanup after exit with status:2 (20200702 11:48:51.815)'
2020-07-02T11:48:51.824943188Z stderr F 2020-07-02 11:48:51,824 - OpenStack-Helm Mariadb - INFO - b"2020-07-02 11:48:51 140031553423104 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address 'mariadb-server-1.mariadb-discovery.openstack.svc.cluster.local' --datadir '/var/lib/mysql/' --parent '2615': 2 (No such file or directory)"
2020-07-02T11:48:51.826103333Z stderr F 2020-07-02 11:48:51,825 - OpenStack-Helm Mariadb - INFO - b'2020-07-02 11:48:51 140031553423104 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.'
2020-07-02T11:48:51.826136549Z stderr F 2020-07-02 11:48:51,826 - OpenStack-Helm Mariadb - INFO - b'2020-07-02 11:48:51 140031884531392 [ERROR] WSREP: SST failed: 2 (No such file or directory)'
2020-07-02T11:48:52.77887252Z stderr F 2020-07-02 11:48:52,778 - OpenStack-Helm Mariadb - INFO - b''
2020-07-02T11:48:52.778945718Z stderr F 2020-07-02 11:48:52,778 - OpenStack-Helm Mariadb - INFO - b'2020-07-02 11:48:51 140031884531392 [ERROR] Aborting'
2020-07-02T11:48:52.779020521Z stderr F 2020-07-02 11:48:52,778 - OpenStack-Helm Mariadb - INFO - b''