2016-08-16 14:00:26 |
Muayad |
description |
We have two servers with Master-Master GTID replication, active-passive mode. After upgrading to 5.7.11,the active master started to run out of memory so often, even with performance schema turned off. By just stopping and starting SLAVE SQL_THREAD, memory usage goes back to normal (innodb buffer pool size + 8% of innodb buffer pool size)
i.e: STOP SLAVE SQL_THREAD;START SLAVE SQL_THREAD
The issue has been reproduced on a test env while active server was running in debug mode(Massif-Valgrind.
how to reproduce:
1- set up master master with GTID replication.
2- create one database; and one table
create table mytest ( id int not null auto_increment primary key, c1 varchar(128), c2 varchar(128) ) engine = innodb;
3- continuously run these queries on it and check RES memory usage,
SELECT count(DISTINCT id) from mytest;
INSERT into mytest (c1,c2) VALUES ('VAL1','VAL2')
UPDATE mytest.mytest set c1='newval1' where id =
attached complete massif.out and globalvariables.out. here is a snippet of massif.out, not sure if pfs_spawn_thread is related to perfomance schema , but mysql instance was started with performance schema off.
->09.32% (578,806,592B) 0x188E91C: my_raw_malloc (in /usr/sbin/mysqld-debug)
| ->09.32% (578,806,592B) 0x188E519: my_malloc (in /usr/sbin/mysqld-debug)
| ->09.03% (561,046,608B) 0x1887A44: alloc_root (in /usr/sbin/mysqld-debug)
| | ->08.75% (543,172,776B) 0xEFC948: Sql_alloc::operator new(unsigned long, st_mem_root*) (in /usr/sbin/mysqld-debug)
| | | ->06.25% (388,424,352B) 0x153B929: LEX::new_query(st_select_lex*) (in /usr/sbin/mysqld-debug)
| | | | ->06.25% (388,424,352B) 0x153BEEE: LEX::new_top_level_query() (in /usr/sbin/mysqld-debug)
| | | | ->06.25% (388,424,352B) 0x153B5E2: lex_start(THD*) (in /usr/sbin/mysqld-debug)
| | | | ->06.25% (388,424,352B) 0x1858CB9: Rpl_info_table_access::before_open(THD*) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x181FA59: System_table_access::open_table(THD*, st_mysql_lex_string, st_mysql_lex_string, unsigned int, thr_lock_type, TABLE**, Open_tables_backup*) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x186E3D3: Rpl_info_table::do_flush_info(bool) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x184BED8: Rpl_info_handler::flush_info(bool) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x1852F79: Relay_log_info::flush_info(bool) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x18502B1: Relay_log_info::inc_group_relay_log_pos(unsigned long long, bool) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x17D0034: Rotate_log_event::do_update_pos(Relay_log_info*) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x184600E: Log_event::update_pos(Relay_log_info*) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x18350C4: apply_event_and_update_pos(Log_event**, THD*, Relay_log_info*) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x1835EC3: exec_relay_log_event(THD*, Relay_log_info*) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x183C23C: handle_slave_sql (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x18CBD18: pfs_spawn_thread (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x4E340A2: start_thread (pthread_create.c:309)
| | | | | ->06.25% (388,424,352B) 0x6E2687B: clone (clone.S:111)
Thanks,
Muayad Yousef |
We have two servers with Master-Master GTID replication, active-passive mode. After upgrading to 5.7.11/5.7.13,the active master started to run out of memory so often, even with performance schema turned off. By just stopping and starting SLAVE SQL_THREAD, memory usage goes back to normal (innodb buffer pool size + 8% of innodb buffer pool size)
i.e: STOP SLAVE SQL_THREAD;START SLAVE SQL_THREAD
The issue has been reproduced on a test env while active server was running in debug mode(Massif-Valgrind.
how to reproduce:
1- set up master master with GTID replication.
2- create one database; and one table
create table mytest ( id int not null auto_increment primary key, c1 varchar(128), c2 varchar(128) ) engine = innodb;
3- continuously run these queries on it and check RES memory usage,
SELECT count(DISTINCT id) from mytest;
INSERT into mytest (c1,c2) VALUES ('VAL1','VAL2')
UPDATE mytest.mytest set c1='newval1' where id =
attached complete massif.out and globalvariables.out. here is a snippet of massif.out, not sure if pfs_spawn_thread is related to perfomance schema , but mysql instance was started with performance schema off.
->09.32% (578,806,592B) 0x188E91C: my_raw_malloc (in /usr/sbin/mysqld-debug)
| ->09.32% (578,806,592B) 0x188E519: my_malloc (in /usr/sbin/mysqld-debug)
| ->09.03% (561,046,608B) 0x1887A44: alloc_root (in /usr/sbin/mysqld-debug)
| | ->08.75% (543,172,776B) 0xEFC948: Sql_alloc::operator new(unsigned long, st_mem_root*) (in /usr/sbin/mysqld-debug)
| | | ->06.25% (388,424,352B) 0x153B929: LEX::new_query(st_select_lex*) (in /usr/sbin/mysqld-debug)
| | | | ->06.25% (388,424,352B) 0x153BEEE: LEX::new_top_level_query() (in /usr/sbin/mysqld-debug)
| | | | ->06.25% (388,424,352B) 0x153B5E2: lex_start(THD*) (in /usr/sbin/mysqld-debug)
| | | | ->06.25% (388,424,352B) 0x1858CB9: Rpl_info_table_access::before_open(THD*) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x181FA59: System_table_access::open_table(THD*, st_mysql_lex_string, st_mysql_lex_string, unsigned int, thr_lock_type, TABLE**, Open_tables_backup*) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x186E3D3: Rpl_info_table::do_flush_info(bool) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x184BED8: Rpl_info_handler::flush_info(bool) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x1852F79: Relay_log_info::flush_info(bool) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x18502B1: Relay_log_info::inc_group_relay_log_pos(unsigned long long, bool) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x17D0034: Rotate_log_event::do_update_pos(Relay_log_info*) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x184600E: Log_event::update_pos(Relay_log_info*) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x18350C4: apply_event_and_update_pos(Log_event**, THD*, Relay_log_info*) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x1835EC3: exec_relay_log_event(THD*, Relay_log_info*) (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x183C23C: handle_slave_sql (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x18CBD18: pfs_spawn_thread (in /usr/sbin/mysqld-debug)
| | | | | ->06.25% (388,424,352B) 0x4E340A2: start_thread (pthread_create.c:309)
| | | | | ->06.25% (388,424,352B) 0x6E2687B: clone (clone.S:111)
Thanks,
Muayad Yousef |
|