Percona Server with XtraDB

malloc(): memory corruption with replication

Reported by PavelVD on 2012-01-13
38
This bug affects 5 people
Affects Status Importance Assigned to Milestone
MySQL Server
Unknown
Unknown
Percona Server
High
George Ormond Lorch III
5.1
Medium
George Ormond Lorch III
5.5
High
George Ormond Lorch III

Bug Description

Since update to 5.5.18
We periodically encounter this error message from the server

*** glibc detected *** /usr/sbin/mysqld: malloc(): memory corruption: 0x00007f8cb104f580 ***
======= Backtrace: =========
/lib/libc.so.6(+0x71ad6)[0x7f930b69bad6]
/lib/libc.so.6(+0x74b6d)[0x7f930b69eb6d]
/lib/libc.so.6(__libc_malloc+0x70)[0x7f930b6a0930]
/usr/sbin/mysqld(my_malloc+0x32)[0x805572]
/usr/sbin/mysqld(alloc_root+0x104)[0x7fdfd4]
/usr/sbin/mysqld(_Z10MYSQLparsePv+0x1836b)[0x66e3db]
/usr/sbin/mysqld(_Z9parse_sqlP3THDP12Parser_stateP19Object_creation_ctx+0x9d)[0x58eb9d]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x262)[0x599012]
/usr/sbin/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0xd5c)[0x73ea0c]
/usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x125)[0x532c65]
/usr/sbin/mysqld[0x537681]
/usr/sbin/mysqld(handle_slave_sql+0xa7b)[0x539bdb]
/lib/libpthread.so.0(+0x68ba)[0x7f930c4578ba]
/lib/libc.so.6(clone+0x6d)[0x7f930b6f902d]
======= Memory map: ========
00400000-00d47000 r-xp 00000000 08:06 32 /usr/sbin/mysqld
00f47000-01063000 rw-p 00947000 08:06 32 /usr/sbin/mysqld
01063000-01090000 rw-p 00000000 00:00 0
01d1c000-2e085000 rw-p 00000000 00:00 0 [heap]
......
7f930a3dc000-7f930a3e7000 r-xp 00000000 08:02 49 /lib/libnss_files-2.11.2.so
7f930a3e7000-7f930a5e6000 ---p 0000b000 08:02 49 /lib/libnss_files-2.11.2.so
7f930a5e6000-7f930a5e7000 r--p 0000a000 08:02 49 /lib/libnss_files-2.11.2.so
7f930a5e7000-7f930a5e8000 rw-p 0000b000 08:02 49 /lib/libnss_files-2.11.2.so
7f930a5e8000-7f930a5f2000 r-xp 00000000 08:02 46 /lib/libnss_nis-2.11.2.so
7f930a5f2000-7f930a7f1000 ---p 0000a000 08:02 46 /lib/libnss_nis-2.11.2.so
7f930a7f1000-7f930a7f2000 r--p 00009000 08:02 46 /lib/libnss_nis-2.11.2.so
7f930a7f2000-7f930a7f3000 rw-p 0000a000 08:02 46 /lib/libnss_nis-2.11.2.so
7f930a7f3000-7f930a808000 r-xp 00000000 08:02 64 /lib/libnsl-2.11.2.so
7f930a808000-7f930aa07000 ---p 00015000 08:02 64 /lib/libnsl-2.11.2.so
7f930aa07000-7f930aa08000 r--p 00014000 08:02 64 /lib/libnsl-2.11.2.so
7f930aa08000-7f930aa09000 rw-p 00015000 08:02 64 /lib/libnsl-2.11.2.so
7f930aa09000-7f930aa0b000 rw-p 00000000 00:00 0
7f930aa0b000-7f930aa12000 r-xp 00000000 08:02 60 /lib/libnss_compat-2.11.2.so
7f930aa12000-7f930ac11000 ---p 00007000 08:02 60 /lib/libnss_compat-2.11.2.so
7f930ac11000-7f930ac12000 r--p 00006000 08:02 60 /lib/libnss_compat-2.11.2.so
7f930ac12000-7f930ac13000 rw-p 00007000 08:02 60 /lib/libnss_compat-2.11.2.so
7f930ac13000-7f930ac29000 r-xp 00000000 08:02 38 /lib/libgcc_s.so.1
7f930ac29000-7f930ae28000 ---p 00016000 08:02 38 /lib/libgcc_s.so.1
7f930ae28000-7f930ae29000 rw-p 00015000 08:02 38 /lib/libgcc_s.so.1
7f930ae29000-7f930ae2a000 ---p 00000000 00:00 0
7f930ae2a000-7f930b62a000 rw-p 00000000 00:00 0
7f930b62a000-7f930b782000 r-xp 00000000 08:02 53 /lib/libc-2.11.2.so
7f930b782000-7f930b981000 ---p 00158000 08:02 53 /lib/libc-2.11.2.so
7f930b981000-7f930b985000 r--p 00157000 08:02 53 /lib/libc-2.11.2.so
7f930b985000-7f930b986000 rw-p 0015b000 08:02 53 /lib/libc-2.11.2.so
7f930b986000-7f930b98b000 rw-p 00000000 00:00 0
7f930b98b000-7f930b98c000 r-xp 00000000 08:02 1063 /lib/libaio.so.1.0.1
7f930b98c000-7f930bb8b000 ---p 00001000 08:02 1063 /lib/libaio.so.1.0.1
7f930bb8b000-7f930bb8c000 rw-p 00000000 08:02 1063 /lib/libaio.so.1.0.1
7f930bb8c000-7f930bb93000 r-xp 00000000 08:02 55 /lib/librt-2.11.2.so
7f930bb93000-7f930bd92000 ---p 00007000 08:02 55 /lib/librt-2.11.2.so
7f930bd92000-7f930bd93000 r--p 00006000 08:02 55 /lib/librt-2.11.2.so
7f930bd93000-7f930bd94000 rw-p 00007000 08:02 55 /lib/librt-2.11.2.so
7f930bd94000-7f930be14000 r-xp 00000000 08:02 62 /lib/libm-2.11.2.so
7f930be14000-7f930c014000 ---p 00080000 08:02 62 /lib/libm-2.11.2.so
7f930c014000-7f930c015000 r--p 00080000 08:02 62 /lib/libm-2.11.2.so
7f930c015000-7f930c016000 rw-p 00081000 08:02 62 /lib/libm-2.11.2.so
7f930c016000-7f930c018000 r-xp 00000000 08:02 51 /lib/libdl-2.11.2.so
7f930c018000-7f930c218000 ---p 00002000 08:02 51 /lib/libdl-2.11.2.so
7f930c218000-7f930c219000 r--p 00002000 08:02 51 /lib/libdl-2.11.2.so
7f930c219000-7f930c21a000 rw-p 00003000 08:02 51 /lib/libdl-2.11.2.so

7f930c21a000-7f930c222000 r-xp 00000000 08:02 66 /lib/libcrypt-2.11.2.so
7f930c222000-7f930c421000 ---p 00008000 08:02 66 /lib/libcrypt-2.11.2.so
7f930c421000-7f930c422000 r--p 00007000 08:02 66 /lib/libcrypt-2.11.2.so
7f930c422000-7f930c423000 rw-p 00008000 08:02 66 /lib/libcrypt-2.11.2.so
7f930c423000-7f930c451000 rw-p 00000000 00:00 0
7f930c451000-7f930c468000 r-xp 00000000 08:02 54 /lib/libpthread-2.11.2.so
7f930c468000-7f930c667000 ---p 00017000 08:02 54 /lib/libpthread-2.11.2.so
7f930c667000-7f930c668000 r--p 00016000 08:02 54 /lib/libpthread-2.11.2.so
7f930c668000-7f930c669000 rw-p 00017000 08:02 54 /lib/libpthread-2.11.2.so
7f930c669000-7f930c66d000 rw-p 00000000 00:00 0
7f930c66d000-7f930c68b000 r-xp 00000000 08:02 68 /lib/ld-2.11.2.so
7f930c6a4000-7f930c6a5000 ---p 00000000 00:00 0
7f930c6a5000-7f930c6d5000 rw-p 00000000 00:00 0
7f930c6d5000-7f930c6d6000 ---p 00000000 00:00 0
7f930c6d6000-7f930c706000 rw-p 00000000 00:00 0
7f930c706000-7f930c707000 ---p 00000000 00:00 0
7f930c707000-7f930c737000 rw-p 00000000 00:00 0
7f930c737000-7f930c738000 ---p 00000000 00:00 0
7f930c738000-7f930c768000 rw-p 00000000 00:00 0
7f930c768000-7f930c769000 ---p 00000000 00:00 0
7f930c769000-7f930c799000 rw-p 00000000 00:00 0
7f930c799000-7f930c79a000 ---p 00000000 00:00 0
7f930c79a000-7f930c7ca000 rw-p 00000000 00:00 0
7f930c7ca000-7f930c7cb000 ---p 00000000 00:00 0
7f930c7cb000-7f930c7fb000 rw-p 00000000 00:00 0
7f930c7fb000-7f930c7fc000 ---p 00000000 00:00 0
7f930c7fc000-7f930c82c000 rw-p 00000000 00:00 0
7f930c82c000-7f930c82d000 ---p 00000000 00:00 0
7f930c82d000-7f930c884000 rw-p 00000000 00:00 0
7f930c887000-7f930c88a000 rw-p 00000000 00:00 0
7f930c88a000-7f930c88b000 r--p 0001d000 08:02 68 /lib/ld-2.11.2.so
7f930c88b000-7f930c88c000 rw-p 0001e000 08:02 68 /lib/ld-2.11.2.so
7f930c88c000-7f930c88d000 rw-p 00000000 00:00 0
7fff6f0a4000-7fff6f0b9000 rw-p 00000000 00:00 0 [stack]
7fff6f1aa000-7fff6f1ab000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
120113 5:43:01 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1073741824
read_buffer_size=2097152
max_used_connections=141
max_threads=400
thread_count=7
connection_count=6
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6787866 K
bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x28814b20
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f930c7f9ea8 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x8094a9]
/usr/sbin/mysqld(handle_segfault+0x3c2)[0x51b402]
/lib/libpthread.so.0(+0xef60)[0x7f930c45ff60]
/lib/libc.so.6(gsignal+0x35)[0x7f930b65c165]
/lib/libc.so.6(abort+0x180)[0x7f930b65ef70]
/lib/libc.so.6(+0x6827b)[0x7f930b69227b]
/lib/libc.so.6(+0x71ad6)[0x7f930b69bad6]
/lib/libc.so.6(+0x74b6d)[0x7f930b69eb6d]
/lib/libc.so.6(__libc_malloc+0x70)[0x7f930b6a0930]
/usr/sbin/mysqld(my_malloc+0x32)[0x805572]
/usr/sbin/mysqld(alloc_root+0x104)[0x7fdfd4]
/usr/sbin/mysqld(_Z10MYSQLparsePv+0x1836b)[0x66e3db]
/usr/sbin/mysqld(_Z9parse_sqlP3THDP12Parser_stateP19Object_creation_ctx+0x9d)[0x58eb9d]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x262)[0x599012]
/usr/sbin/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0xd5c)[0x73ea0c]
/usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x125)[0x532c65]
/usr/sbin/mysqld[0x537681]
/usr/sbin/mysqld(handle_slave_sql+0xa7b)[0x539bdb]
/lib/libpthread.so.0(+0x68ba)[0x7f930c4578ba]
/lib/libc.so.6(clone+0x6d)[0x7f930b6f902d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f8cb104f128): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
mysqld: /home/jenkins/workspace/percona-server-5.5-debs/label_exp/debian5-64/target/Percona-Server-5.5.18-rel23.0/mysys/my_new.cc:52: int __cxa_pure_virtual(): Assertion `! "Aborted: pure virtual method called."'
 failed.

query_cache_strip_comments is off
+------------------------------+----------+
| Variable_name | Value |
+------------------------------+----------+
| have_query_cache | YES |
| query_cache_limit | 2097152 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 50331648 |
| query_cache_strip_comments | OFF |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
+------------------------------+----------+

Alexey Kopytov (akopytov) wrote :

Looks exactly like the 5.5 counterpart of bug #908531.

Alexey Kopytov (akopytov) wrote :

This is reported against 5.5.18 which has the fix for bug #705688. So it cannot be a duplicate of bug #705688. Restoring the original status.

Oleg Tsarev (tsarev) wrote :

Can you please provide following information:

1) Exact query which acquired the crash (if you have specific)
2) Binary log which acquired the crash with some events before
3) Schema of your tables (if this possible, of course)

So, if this problem related to query cache enhance then disable query cache will help you.

Changed in percona-server:
assignee: nobody → Oleg Tsarev (tsarev)
importance: Undecided → High
status: New → Incomplete
PavelVD (pdobryakov) wrote :

At the moment I do not have the opportunity to provide excerpts from bin-logs, as I rolled all the problematic server to version 5.5.17 after the occurrence of this problem.
Indicate the particular query was not possible, unfortunately, because We use mixed and row based replication, and bin logs was difficult to say what caused the fall.
We have encountered this problem on about 20 different servers with very different databases and schemas.
The first thing I did when faced with this problem - I disabled the query cache, but this had no impact on the situation.

Oleg Tsarev (tsarev) on 2012-02-02
Changed in percona-server:
assignee: Oleg Tsarev (tsarev) → Patrick Crews (patrick-crews)
Oleg Tsarev (tsarev) wrote :

Statement Based Replication doesn't send to slave "just SELECT" queries.
Query Cache works with "just SELECT" queries (other are ignored).

Related to query-cache-strip-comments code doesn't allocate memory if
query "not a SELECT".

From this point of view I'm sure - this problems not related to
query-cache-strip-comments,

PavelVD (pdobryakov) wrote :
Download full text (10.5 KiB)

query on master:

USE `sas-ru`;

CREATE TABLE analytics_sections_categories(
  id INT(11) NOT NULL AUTO_INCREMENT,
  section_id SMALLINT(6) NOT NULL,
  category_id SMALLINT(6) NOT NULL,
  PRIMARY KEY (section_id, category_id),
  INDEX FK_analytics_category (category_id),
  UNIQUE INDEX id (id),
  CONSTRAINT FK_analytics_category FOREIGN KEY (category_id)
  REFERENCES analytics_category (id) ON DELETE RESTRICT ON UPDATE RESTRICT,
  CONSTRAINT FK_analytics_section FOREIGN KEY (section_id)
  REFERENCES analytics_section (id) ON DELETE RESTRICT ON UPDATE RESTRICT
)
ENGINE = INNODB
CHARACTER SET utf8
COLLATE utf8_general_ci;

USE `site-ru-v4`;
CREATE TABLE analytics_sections_categories(
  id INT(11) NOT NULL AUTO_INCREMENT,
  section_id SMALLINT(6) NOT NULL,
  category_id SMALLINT(6) NOT NULL,
  PRIMARY KEY (section_id, category_id),
  INDEX FK_analytics_category (category_id),
  UNIQUE INDEX id (id),
  CONSTRAINT FK_analytics_category FOREIGN KEY (category_id)
  REFERENCES analytics_category (id) ON DELETE RESTRICT ON UPDATE RESTRICT,
  CONSTRAINT FK_analytics_section FOREIGN KEY (section_id)
  REFERENCES analytics_section (id) ON DELETE RESTRICT ON UPDATE RESTRICT
)
ENGINE = INNODB
CHARACTER SET utf8
COLLATE utf8_general_ci;

I`ve emulate those situation and get error. Here is records in binlog
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#120203 5:25:06 server id 125009144 end_log_pos 107 Start: binlog v 4, server v 5.5.18-55-log created 120203 5:25:06
# Warning: this binlog is either in use or was not closed properly.
BINLOG '
klMrTw/4fHMHZwAAAGsAAAABAAQANS41LjE4LTU1LWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAEzgNAAgAEgAEBAQEEgAAVAAEGggAAAAICAgCAA==
'/*!*/;
# at 555729637
#120203 13:23:00 server id 125009144 end_log_pos 555729859 Query thread_id=3657294 exec_time=0 error_code=0
SET TIMESTAMP=1328268180/*!*/;
SET @@session.pseudo_thread_id=3657294/*!*/;
SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;
SET @@session.sql_mode=0/*!*/;
SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;
/*!\C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=33/*!*/;
SET @@session.lc_time_names=0/*!*/;
SET @@session.collation_database=DEFAULT/*!*/;
UPDATE `phpmyadmin`.`pma_tracking` SET `tracking_active` = '0' WHERE `db_name` = 'sas-ru' AND `table_name` = 'analytics_sections_categories' AND `version` = '0'
/*!*/;
# at 555729859
#120203 13:23:00 server id 125009144 end_log_pos 555729924 Query thread_id=3657294 exec_time=0 error_code=0
SET TIMESTAMP=1328268180/*!*/;
COMMIT
/*!*/;
# at 555729924
#120203 13:23:01 server id 125009144 end_log_pos 555730600 Query thread_id=3657295 exec_time=0 error_code=0
use site-ru-v4/*!*/;
SET TIMESTAMP=1328268181/*!*/;
CREATE TABLE analytics_sections_categories(
  id INT(11) NOT NULL AUTO_INCREMENT,
  section_id SMALLINT(6) NOT NULL,
  category_id SMALLINT(6) NOT NULL,
  PRIMARY KEY (section_id, ca...

PavelVD (pdobryakov) wrote :

What other information necessary to identify the problem?

PavelVD (pdobryakov) wrote :

there is also a bug in version 5.5.20

Dima (l-dima) wrote :

I have recently upgraded to Percona 5.5.20 and I am hitting this bug every 2-3 days.
My instances which hit that are slave-only instances for backup purposes. MySQL locks up, cannot be shutdown and has to be killed.

If somebody interested in pursuing this bug let me know what info you need me to collect and how to do it.

Thanks,
D.

PavelVD (pdobryakov) wrote :

There, do you have plans to address this bug?
Because of this error, we can not update the version 5.5.18 or older at all our production servers that have extremely nice.

PavelVD (pdobryakov) wrote :

Why does the error status is "incomplete" to 2012-02-01?
What additional information is needed to fix this error?

Vadim Tkachenko (vadim-tk) wrote :

PavelVD,

We will look into this, and let you know if additional information is needed.

Changed in percona-server:
status: Incomplete → New
Alexey Kopytov (akopytov) wrote :

Pavel,

I could not reproduce the crash in my environment using the queries you provided in comment #6, because "CREATE TABLE analytics_sections_categories..." fails to resolve the analytics_section table in an FK constraint.

You mentioned that you've emulated the error. Does this mean you have an isolated test case reproducing the crash?

Changed in percona-server:
status: New → Incomplete
PavelVD (pdobryakov) wrote :

Unfortunately, I do not have possibility to create a stand-alone test because in other cases where there was the collapse of servers in the bin log entries were different.

PavelVD (pdobryakov) wrote :

Now, I checked the logs after a further 3 falls - no strange entries in the relay-log and in the bin-logs were not.

PavelVD (pdobryakov) wrote :

Bin-logs on master at that moment

PavelVD (pdobryakov) wrote :

error logs on server at that moment

Changed in percona-server:
assignee: Patrick Crews (patrick-crews) → nobody
Changed in percona-server:
milestone: none → 5.5.21-25.0
assignee: nobody → Alexey Kopytov (akopytov)
status: Incomplete → In Progress
Changed in percona-server:
status: In Progress → Fix Committed
Steffen Boehme (boemm) wrote :

It seems for me, that percona offers stable releases once a month, so the next release may be provided in the middle of April ... am I right?
In this case we had to rollback our servers to the last version not affected by this bug.

Stewart Smith (stewart) on 2012-03-23
Changed in percona-server:
milestone: 5.5.21-25.0 → 5.5.21-25.1
Alexey Kopytov (akopytov) wrote :

Steffen,

We are going to provide 5.5.21-25.1 with a fix for this bug next week.

Steffen Boehme (boemm) wrote :

That great, thanks for your support!

Alexey Kopytov (akopytov) wrote :

We didn't have a reproducible test case for this bug. So we just reverted non-essential refactoring introduced in 5.5.18 as the most likely culprit.

It turns out it's an upstream issue after all: http://bugs.mysql.com/bug.php?id=64624, as I originally thought. At least the symptoms and affected versions are the same.

This has been fixed upstream in 5.5.26 (and is blocking, which has kept us on Percona 5.5.16). Reference: http://bugs.mysql.com/bug.php?id=64624

Any chance we'll see this soon in Percona?

On Tue, 12 Jun 2012 23:17:56 -0000, "Michael S. Moody" <email address hidden> wrote:
> This has been fixed upstream in 5.5.26 (and is blocking, which has kept
> us on Percona 5.5.16). Reference: http://bugs.mysql.com/bug.php?id=64624
>
> Any chance we'll see this soon in Percona?

We commit to having a Percona Server release for each new MySQL point
release within 1 month of the MySQL release.

--
Stewart Smith

Stewart Smith (stewart) wrote :

On Tue, 12 Jun 2012 23:17:56 -0000, "Michael S. Moody" <email address hidden> wrote:
> This has been fixed upstream in 5.5.26 (and is blocking, which has kept
> us on Percona 5.5.16). Reference: http://bugs.mysql.com/bug.php?id=64624
>
> Any chance we'll see this soon in Percona?

sorry, I spoke too soon, not looking at the bug number. we have already
provided a fix in the latest Percona Server release: 5.5.24-26.0

--
Stewart Smith

summary: - malloc(): memory corruption with 5.5.18
+ malloc(): memory corruption with replication
fengyi (fengyi) wrote :

Master : 5.1.48
Slave : 5.5.18(query cache on)

as http://bugs.mysql.com/bug.php?id=64624 described, I run the slave in valgrind, and got following infomation:

Thread 21:
Conditional jump or move depends on uninitialised value(s)
    at 0x5A6743: Query_cache::send_result_to_client(THD*, char*, unsigned int) (sql_cache.cc:2051)
    by 0x5EDE27: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:5756)
    by 0x800B79: Query_log_event::do_apply_event(Relay_log_info const*, char const*, unsigned int) (log_event.cc:3398)
    by 0x800157: Query_log_event::do_apply_event(Relay_log_info const*) (log_event.cc:3166)
    by 0x572005: Log_event::apply_event(Relay_log_info const*) (log_event.h:1135)
    by 0x56B1A9: apply_event_and_update_pos(Log_event*, THD*, Relay_log_info*) (slave.cc:2351)
    by 0x56B6E9: exec_relay_log_event(THD*, Relay_log_info*) (slave.cc:2511)
    by 0x56D8F6: handle_slave_sql (slave.cc:3329)
    by 0x3A01806D63: start_thread (pthread_create.c:308)

I looked into the souce code, and find that the query buffer allocated in Query_log_event::Query_log_event does not contain the db_len field : sizeof(size_t), but in alloc_query there are statement, length, db_name and flag, therefore result in uninitialized data access in Query_cache::send_result_to_client :

size_t db_len;
    memcpy((char *) &db_len, (sql + query_length + 1), sizeof(size_t));
    if (thd->db_length != db_len)
    {

but I didn't found that this can lead to server crash as the uninitialized data are just junk, but not NULL pointer.

can anyone can tell me why this bug can crash slave?

Hui Liu (hickey) wrote :

Hi Fengyi, the slave is crashed due to the memory mistaken of addresses are ruined. Just like this issue, after long time running, mysqld might crashed here and there, though you will see the top backtrace is malloc() or /free().

Hui Liu (hickey) wrote :

Instead of alloc a new buf and then copy all content with the correct format in Query_log_event::do_apply_event(), Oracle alloc the extra db_len and then copy the size to get the matched buf in Query_log_event::Query_log_event().

Patch could ref to http://bazaar.launchpad.net/~mysql/mysql-server/5.5/revision/3837

They are opposite though to solve this problem.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.