Slave SQL Thread run out of memeory and Server crashed

Bug #1714746 reported by jixianliang
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
New
Undecided
Unassigned

Bug Description

Server version: Percona-Server-5.7.18-16
Audit_log plugin: Enabled
Master Server: Binlog_format=Satatement

Slave server crashed with signal 11 in function 'realloc_stack_frames', due to 'SQL Thread running out of memeory'. The frame 'local->stack.frames' becomes biger and biger as the SQL thread runs the statements coming from the Master Server, eventually the slave server runs out of memory. The following is the backtrace:

Core was generated by `/usr/mtdp_mysql/install/bin/mysqld --basedir=/usr/mtdp_mysql/install --datadir='.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f138b73897c in pthread_kill () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6_7.7.x86_64 jansson-2.6-1.el6.x86_64 libgcc-4.4.7-18.el6.x86_64 libstdc++-4.4.7-18.el6.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x00007f138b73897c in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000007c3e44 in handle_fatal_signal (sig=11) at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/signal_handler.cc:223
#2 <signal handler called>
#3 0x00007f138a423359 in __memset_sse2 () from /lib64/libc.so.6
#4 0x00007f1143f2f15c in realloc_stack_frames (thd=0x7f10c800a140, size=134217728) at /usr/include/bits/string3.h:85
#5 0x00007f1143f30cb3 in audit_log_update_thd_local (thd=0x7f10c800a140, event_class=MYSQL_AUDIT_TABLE_ACCESS_CLASS, event=0x7f113c144900)
    at /home/jixianliang/percona_git/percona-server-5.7.18-16/plugin/audit_log/audit_log.c:1189
#6 audit_log_notify (thd=0x7f10c800a140, event_class=MYSQL_AUDIT_TABLE_ACCESS_CLASS, event=0x7f113c144900)
    at /home/jixianliang/percona_git/percona-server-5.7.18-16/plugin/audit_log/audit_log.c:1220
#7 0x00000000007c527f in plugins_dispatch (thd=0x7f10c800a140, event_class=MYSQL_AUDIT_TABLE_ACCESS_CLASS, event_name=0x134ed40 "MYSQL_AUDIT_TABLE_ACCESS_UPDATE",
    event=0x7f113c144900) at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/sql_audit.cc:1370
#8 event_class_dispatch (thd=0x7f10c800a140, event_class=MYSQL_AUDIT_TABLE_ACCESS_CLASS, event_name=0x134ed40 "MYSQL_AUDIT_TABLE_ACCESS_UPDATE", event=0x7f113c144900)
    at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/sql_audit.cc:1410
#9 event_class_dispatch_error (thd=0x7f10c800a140, event_class=MYSQL_AUDIT_TABLE_ACCESS_CLASS, event_name=0x134ed40 "MYSQL_AUDIT_TABLE_ACCESS_UPDATE",
    event=0x7f113c144900) at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/sql_audit.cc:1432
#10 0x00000000007c57b5 in mysql_audit_notify (thd=0x7f10c800a140, subclass=<value optimized out>, subclass_name=0x134ed40 "MYSQL_AUDIT_TABLE_ACCESS_UPDATE",
    table=0x7f10c8010050) at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/sql_audit.cc:624
#11 0x00000000007c58a1 in mysql_audit_table_access_notify (thd=0x7f10c800a140, table=0x7f10c8010050)
    at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/sql_audit.cc:702
#12 0x0000000000cbd502 in open_tables (thd=0x7f10c800a140, start=0x7f113c144af8, counter=0x7f10c800c5e8, flags=0, prelocking_strategy=0x7f113c144b10)
    at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/sql_base.cc:5841
#13 0x0000000000cbe4e2 in open_tables_for_query (thd=0x7f10c800a140, tables=0x7f10c8010050, flags=<value optimized out>)
    at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/sql_base.cc:6568
#14 0x0000000000da1c27 in Sql_cmd_update::try_single_table_update (this=0x7f10c8010018, thd=0x7f10c800a140, switch_to_multitable=0x7f113c144bef)
    at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/sql_update.cc:2870
#15 0x0000000000da1e1c in Sql_cmd_update::execute (this=0x7f10c8010018, thd=0x7f10c800a140)
    at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/sql_update.cc:3017
#16 0x0000000000d14386 in mysql_execute_command (thd=0x7f10c800a140, first_level=true) at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/sql_parse.cc:3796
#17 0x0000000000d18435 in mysql_parse (thd=0x7f10c800a140, parser_state=<value optimized out>)
    at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/sql_parse.cc:5929
#18 0x0000000000eccf43 in Query_log_event::do_apply_event (this=0x7f10bc103210, rli=0x7f10bc02cef0,
    query_arg=0x7f10bc104053 "update host_list set log_agent_file_version = 'log_agent_file-v1.2.3', log_agent_file_last_heartbeat = '2017-08-29 10:56:49' where host = 'dengine-web-a06.yp'", q_len_arg=139710051696472) at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/log_event.cc:4642
#19 0x0000000000f20db5 in slave_worker_exec_job_group (worker=0x7f10bc02cef0, rli=0x3c5c4d0)
    at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/rpl_rli_pdb.cc:2710
#20 0x0000000000f096f3 in handle_slave_worker (arg=0x7f10bc02cef0) at /home/jixianliang/percona_git/percona-server-5.7.18-16/sql/rpl_slave.cc:6203
#21 0x0000000000f5b141 in pfs_spawn_thread (arg=0x7f10bc030ec0) at /home/jixianliang/percona_git/percona-server-5.7.18-16/storage/perfschema/pfs.cc:2188
#22 0x00007f138b733aa1 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f138a48793d in clone () from /lib64/libc.so.6

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3738

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.