Percona Server with XtraDB

mysqld crash double free or corruption

Reported by Billy Becker on 2011-01-21
40
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Percona Server
High
Unassigned
5.1
High
Unassigned
5.5
High
Unassigned

Bug Description

We're running Percona-Server-server-51-5.1.47-rel11.2.53.rhel5 on centos 5.5 x64

This system has 2 quad-core cpus ( xeon 5620 ) and 72GB ram

the issue happened a couple times this morning.

we're not sure which query it was running when the crash happened.

I'm attaching the portion of the error log with the backtrace for these crashes

It looks there was a double free in the QueryStripComments class cleanup method?

After the initial crash, there was another crash after it finished the recovery.

It restarted after that crash, and then it crashed again, again in the QueryStripComments cleanup method

the server started again after that crash, and we logged onto mysql and set query_cache_strip_comments to 0, and the system has been running fine since then.

Billy Becker (billy-becker) wrote :
Stewart Smith (stewart) wrote :

Are you seeing this with current Percona Server releases? I've done a quick audit of the code and this does not currently seem possible. We can reopen this bug if you can reproduce with current Percona Server release.

Thanks for your bug report,
stewart

Changed in percona-server:
status: New → Incomplete
koyaSreekath (koya-sreekanth) wrote :

We also have same issue with this parameter and setting to query_cache_strip_comments to 0 mysql port looks stable now.

I am using Percona Server Version:
Server version: 5.1.58rel12.9-log Percona Server with XtraDB (GPL), Release rel12.9, Revision 271

Appreciate if we can get some quick feedback

thanks.
sreekanth.

Oleg Tsarev (tsarev) on 2011-10-24
Changed in percona-server:
assignee: nobody → Oleg Tsarev (tsarev)
Dr. Tyrell (dr.tyrell) wrote :

Same here, Percona-Server-server-51.x86_64 5.1.59-rel13.0.325.rhel5
I have turned off query_cache_strip_comments based on this thread and also 856404

Alexey Kopytov (akopytov) wrote :

I don't understand the status of this bug. Is this a duplicate of bug #856404? If so, this one should be marked a duplicate.

But I don't see any proof or explanations it's a duplicate. We don't even have a test case for it.

Oleg Tsarev (tsarev) wrote :

Alexey, this is another bug related to query cache enhance (also fixed in my MP).

Alexey Kopytov (akopytov) wrote :

I don't see this bug mentioned anywhere in revision comments in the fix for bug #856404.

Can we have a problem description for this bug then? Like what queries can lead to a crash under what circumstances? To make it searchable and to make sure we have this case covered in the bugfix and tests?

Alexey Kopytov (akopytov) wrote :

I would like to see an evidence this bug is actually fixed by https://code.launchpad.net/~tsarev/percona-server/5.5_fix_bug_856404/+merge/86003. Test cases, technical explanation of the root cause, whatever.

Oleg Tsarev (tsarev) wrote :

Alexey,

Than original fix was in commit 192.2.1:
    revno: 192.2.1
    fixes bug(s): https://launchpad.net/bugs/856404
    committer: Oleg Tsarev <email address hidden>
    branch nick: query_cache_strip_comments
    timestamp: Tue 2011-11-08 22:10:56 +0300
    message:
      This commit contain test and fix for buf #856404.
      While I am allocating buffer for striped query I forgot about \0 in the end of query.
      After that memory is corrupted.

@@ -108,7 +108,8 @@
 +}
 +void QueryStripComments::set(const char* query, uint query_length, uint additional_length)
 +{
-+ uint new_buffer_length = query_length + additional_length;
++ /* Don't forget about \0 in the end of query */
++ uint new_buffer_length = query_length + 1 + additional_length;
 + if(new_buffer_length > buffer_length)
 + {
 + cleanup();

Without this I receive error "double free of curruption".
After correct allocation error go out.

Do you need some another explanations?

Oleg Tsarev (tsarev) wrote :

Alexey,

Also this bug was reproduced by test in: https://code.launchpad.net/~tsarev/dbqp/dbqp_out_of_source

We haven't reproduced this bug on 5.1, thus marking as "Incomplete." If you experience server crashes with query cache comment stripping enabled on 5.1 (or 5.5), please reopen. Thanks!

Alexey Kopytov (akopytov) wrote :

@Alejandra Bedolla Diaz,

Please don't change the Status and Assignment fields in the Percona Server project. Those are used to track bug states in our project. If this affects another project and you want to change the state for that specific project, please use the "Also affects project" link and change the bug state there.

The crash was seen on 5.1, re-opening this bug for 5.1 to backport https://code.launchpad.net/~akopytov/percona-server/bug915814-5.5 revision 228 fix.

Vlad Lesin (vlad-lesin) wrote :

Laurynas, the fix for the issue was merged to 5.1 in revision number 443. The differences between Alexey's fix and fix in 443 revision are not so big and described and backported here lp:~vlad-lesin/percona-server/5.1-bug915814 .

The main issue of wrong buffer size calculating and allocating was solved in revision 443. The following change in "void QueryStripComments::set(const char* query, uint query_length, uint additional_length)" could help to avoid crashes in 5.1 tied with query_cache_strip_comments option:

@@ -466,7 +466,7 @@
         buffer[position++] = ' ';
       }
     }
- else
+ else if (query_position < query_length)
     {
       buffer[position++] = query[query_position++];
     }

Otherwise we faced with another bug.

Vlad, thanks, indeed there were only minor bits to backport, I was misled by the lack of a testcase, missing QUERY_BUFFER_ADDITIONAL_LENGTH macro, and because I didn't see 5.1 rev 443.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers