Slave server crash after an update statement
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| MySQL Server |
Unknown
|
Unknown
|
||
| Percona Server moved to https://jira.percona.com/projects/PS |
Fix Released
|
High
|
George Ormond Lorch III | |
| 5.5 |
Fix Released
|
High
|
George Ormond Lorch III |
Bug Description
The latest 5.5.27-28.1 Percona Server crashes when the following list of things happens:
- execute a update statement using a variable
- slave server has replicate-
MASTER: 5.5.24-rel26.0-log
(default sandbox my.cnf)
[mysql]
prompt='mysql [\h] {\u} (\d) > '
#
[client]
user = msandbox
password = msandbox
port = 5524
socket = /tmp/mysql_
[mysqld]
user = root
port = 5524
socket = /tmp/mysql_
basedir = /root/5.5.24
datadir = /root/sandboxes
tmpdir = /root/sandboxes
pid-file = /root/sandboxes
#log-slow-queries = /root/sandboxes
#log = /root/sandboxes
#
# additional options passed through 'my_clause'
#
log-error=
server_id=1
log-bin=a
SLAVE: 5.5.27-28.1 Percona Server
root@debian:
[mysqld]
datadir=
innodb_
innodb_
server_id=2
replicate-
replicate-
master> create database mytest;
master> use mytest;
master> create table t (i int, a char(10), b char(11));
master> insert into t values (1,'xxx','yyy');
master> update mytest.t set a='xxx'
then, restart the MASTER:
master> use mytest;
master> update mytest.t set i=1;
master> update mytest.t set a='xxx'
then, on the SLAVE:
slave> stop slave;
slave> start slave;
slave> show slave status\G
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/
ERROR:
Can't connect to the server
1:04:07 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://
key_buffer_
read_buffer_
max_used_
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x2b04580
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f9e92a59ea8 thread_stack 0x40000
/usr/sbin/
/usr/sbin/
/lib/libpthread
/lib/libc.
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/libpthread
/lib/libc.
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (2b436a0): ?6?
Connection ID (thread ID): 1
Status: NOT_KILLED
You may download the Percona Server operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
Related branches
- Laurynas Biveinis: Approve on 2012-10-11
-
Diff: 11 lines (+1/-1)1 file modifiedPercona-Server/sql/log_event.h (+1/-1)
tags: | added: i25112 |
tags: | added: replication |
summary: |
- Server crash after an update statement + Slave server crash after an update statement |
description: | updated |
Finally, I've found it. Some restarts are required to trigger the bug. Description updated.
description: | updated |
The workaround is to disable the following parameters:
replicate-
replicate-
and then mysql starts to work again.
Submitted a bug report to Oracle:
The slave binary is from 64-bit squeeze deb.
The top of the stacktrace refers to
if (strcmp(alias, tl->alias))
at line 3337 in TABLE::init(THD *, TABLE_LIST *)
MTR test case:
[openxs@chief mysql-test]$ cat t/bug65831.test source include/
--echo **** Master ****
connection master;
create database mytest;
use mytest;
CREATE TABLE t (a int, b int, c int);
INSERT INTO t VALUES(1, 2, 3);
UPDATE mytest.t set b=1, c=1, a=@var:=@var+1 where a=1;
SELECT * FROM t;
UPDATE t set a=1;
UPDATE mytest.t set b=1, c=1, a=@var:=@var+1 where a=1;
--echo **** Slave ****
# sync_slave_
connection slave;
stop slave;
start slave;
show slave status;
# Cleanup
connection master;
DROP TABLE t;
sync_slave_
Tom McCann (09-tom-an) wrote : | #7 |
Not sure if the crash I'm getting is the same. We do have a couple replicate-
22:21:16 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://
key_buffer_
read_buffer_
max_used_
max_threads=500
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0xb0cfa90
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f6a102d8838 thread_stack 0x40000
/usr/sbin/
/usr/sbin/
/lib/libpthread
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/libpthread
/lib/libc.
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (b0c7df0): `??
Connection ID (thread ID): 27
Status: NOT_KILLED
Roel Van de Paar (roel11) wrote : | #8 |
For clarity, the original stacktrace cleaned up by running it through c++filt:
/usr/sbin/
/usr/sbin/
/lib/libpthread
/lib/libc.
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
long, enum_duplicates, bool, unsigned long long*, unsigned long long*)+
ommand(
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/libpthread
/lib/libc.
And, the stacktrace from #7 cleaned up:
/usr/sbin/
/usr/sbin/
/lib/libpthread
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib/libpthread
/lib/libc.
The second stack trace is a different bug, which may or may not be related to bug #1042946 (it's a different stack, but it's reported as a Valgrind issue for the moment, so the crashing stack may be different).
Tom, if you are able to create a testcase, or provide us with a core dump + mysqld binary, it would help tremendously. In any case, a separate bug report should be logged for this.
Not sure if this helps, but this attached is a copy of mysqld crashing when I was setting up a new slave.
On Nov 28, 2012, at 9:31 PM, RoelV wrote:
> For clarity, the original stacktrace cleaned up by running it through
> c++filt:
>
> /usr/sbin/
> /usr/sbin/
> /lib/libpthread
> /lib/libc.
> /usr/sbin/
> /usr/sbin/
> /usr/sbin/
> /usr/sbin/
> long, enum_duplicates, bool, unsigned long long*, unsigned long long*)+
> ommand(
> /usr/sbin/
> /usr/sbin/
> /usr/sbin/
> /usr/sbin/
> /usr/sbin/
> /lib/libpthread
> /lib/libc.
>
> And, the stacktrace from #7 cleaned up:
>
> /usr/sbin/
> /usr/sbin/
> /lib/libpthread
> /usr/sbin/
> /usr/sbin/
> /usr/sbin/
> /usr/sbin/
> /usr/sbin/
> /usr/sbin/
> /usr/sbin/
> /lib/libpthread
> /lib/libc.
>
> The second stack trace is a different bug, which may or may not be
> related to bug #1042946 (it's a different stack, but it's reported as a
> Valgrind issue for the moment, so the crashing stack may be different).
>
> Tom, if you are able to create a testcase, or provide us with a core
> dump + mysqld binary, it would help tremendously. In any case, a
> separate bug report should be logged for this.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https:/
>
> Title:
> Slave server crash after an update statement
>
> Status in MySQL Server:
> Unknown
> Status in Percona Server with XtraDB:
> Fix Released
> Status in Percona Server 5.5 series:
> Fix Released
>
> Bug description:
> The latest 5.5.27-28.1 Percona Server crashes when the following list
> of things happens:
>
> - execute a update statemen...
Roel Van de Paar (roel11) wrote : | #10 |
Tom; very interesting. Is there any possibility for us to get a copy of MASTER_
Tom McCann (09-tom-an) wrote : | #11 |
Attached is a zip with 3 binlogs starting from 78. If there was a core dump where would it be (on an Ubuntu 10.04 EC2 system)? The process I'm using to bring up new slaves may be flawed and is what's causing this. I start a new EC2 instance from an AMI I made a few months ago (has existing mysql data on it). I bring over a xtrabackup from the master and restore. When I start the slave after the restore (change master/start slave), it crashes almost immediately.
I've since downgraded this system to 5.5.24-55 but I can bring up another instance and give you access to it if that helps.
On Dec 5, 2012, at 2:57 PM, RoelV wrote:
> Tom; very interesting. Is there any possibility for us to get a copy of
> MASTER_
> Was there any date on the slave already (copied?)? Lastly, was a core
> file generated? How large is it when compressed + mysqld included? Could
> we obtain the same?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https:/
>
> Title:
> Slave server crash after an update statement
>
> Status in MySQL Server:
> Unknown
> Status in Percona Server with XtraDB:
> Fix Released
> Status in Percona Server 5.5 series:
> Fix Released
>
> Bug description:
> The latest 5.5.27-28.1 Percona Server crashes when the following list
> of things happens:
>
> - execute a update statement using a variable
> - slave server has replicate-
>
> MASTER: 5.5.24-rel26.0-log
>
> (default sandbox my.cnf)
>
> [mysql]
> prompt='mysql [\h] {\u} (\d) > '
> #
>
> [client]
> user = msandbox
> password = msandbox
> port = 5524
> socket = /tmp/mysql_
>
> [mysqld]
> user = root
> port = 5524
> socket = /tmp/mysql_
> basedir = /root/5.5.24
> datadir = /root/sandboxes
> tmpdir = /root/sandboxes
> pid-file = /root/sandboxes
> #log-slow-queries = /root/sandboxes
> #log = /root/sandboxes
> #
> # additional options passed through 'my_clause'
> #
> log-error=
> server_id=1
> log-bin=a
>
> SLAVE: 5.5.27-28.1 Percona Server
>
> root@debian:
> [mysqld]
> datadir=
> innodb_
> innodb_
> server_id=2
> replicate-
> replicate-
>
> master> create database mytest;
> master> use mytest;
> master> create table t (i int, a char(10), b char(11));
> master> insert into t values (1,'xxx','yyy');
> master> update mytest.t set a='xxx'
>
> then, restart the MASTER:
>
> master> use mytest;
> master> update mytest.t set i=1;
> master> update mytest.t set a='xx...
Percona now uses JIRA for bug reports so this bug report is migrated to: https:/
I've reinstalled everything again and now following the same procedure it doesn't crash, seems that something is messing.