PXC 5.5.30-23.7.4.404 not able the use a different shared memory allocator library

Bug #1170345 reported by Frederic Descamps on 2013-04-18
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
Medium
Alex Yurchenko
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Fix Released
Undecided
Raghavendra D Prabhu

Bug Description

With the latest version of PXC, it's not possible to define a malloc-lib and start the server using init script or mysqld_safe.

To define the malloc-lib in my.cnf:

[mysqld_safe]
malloc-lib=/usr/lib64/libjemalloc.so.1

Then when you start using init script, this is the only message you get in error log:

30418 07:13:08 mysqld_safe Starting mysqld daemon with databases from /data/mysql
130418 07:13:08 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.AmupX31802
130418 07:13:08 mysqld_safe WSREP: Failed to recover position:

When you start it manually with mysld_safe:

130418 07:22:50 mysqld_safe Adding '/usr/lib64/libjemalloc.so.1' to LD_PRELOAD for mysqld
130418 07:22:50 mysqld_safe Logging to '/data/mysql/mysql-error.log'.
130418 07:22:50 mysqld_safe Starting mysqld daemon with databases from /data/mysql
130418 07:22:50 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.JvKKr11395
/usr/bin/mysqld_safe: line 224: LD_PRELOAD=/usr/lib64/libjemalloc.so.1: No such file or directory
130418 07:22:50 mysqld_safe WSREP: Failed to recover position:

The command launched is : LD_PRELOAD=/usr/lib64/libjemalloc.so.1 nohup /usr/sbin/mysqld --basedir=/usr --datadir=/data/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/data/mysql/mysql-error.log --open-files-limit=8192 --pid-file=/data/mysql/mysql.pid --socket=/data/mysql/mysql.sock --log_error=/tmp/tmp.MXrZjr5440 --wsrep-recover

How to fix ?

on line 224 of /usr/bin/mysqld_safe replace

$mysqld_cmd --log_error=$wr_logfile --wsrep-recover

by:

eval $mysqld_cmd --log_error=$wr_logfile --wsrep-recover

summary: - PXC 5.5.30-23.7.4.404 not able the use a different share memory
- allocatot library
+ PXC 5.5.30-23.7.4.404 not able the use a different shared memory
+ allocator library

Yes, the wsrep_recovery needs eval_log_error:

diff -u =mysqld_safe /tmp/mysqld_safe
--- /usr/bin/mysqld_safe 2013-04-18 18:48:27.595069642 +0530
+++ /tmp/mysqld_safe 2013-04-18 18:45:55.539099781 +0530
@@ -221,7 +221,7 @@

   log_notice "WSREP: Running position recovery with --log_error=$wr_logfile"

- $mysqld_cmd --log_error=$wr_logfile --wsrep-recover
+ eval_log_error "$mysqld_cmd --log_error=$wr_logfile --wsrep-recover"

   local rp="$(grep 'WSREP: Recovered position:' $wr_logfile)"
   if [ -z "$rp" ]; then

I don't think this is a merge regression, this bug has been present ever since wsrep_recover was added to mysqld_safe.

But I presume nobody tried to use malloc-lib with PXC before, so it is failing now.

Another workaround would be to export LD_PRELOAD in the init script.

Changed in percona-xtradb-cluster:
status: New → Confirmed
assignee: nobody → Raghavendra D Prabhu (raghavendra-prabhu)
Changed in codership-mysql:
assignee: nobody → Alex Yurchenko (ayurchen)
importance: Undecided → Medium
milestone: none → 5.5.30-24.8
status: New → Confirmed
Frederic Descamps (lefred) wrote :

Was working with Percona-XtraDB-Cluster-server-5.5.29-23.7.1.388

Changed in percona-xtradb-cluster:
status: Confirmed → Fix Committed

No, it wasn't.

==========================================================
+ chmod 600 /tmp/tmp.VU380MxLV9
+ log_notice WSREP: Running position recovery with --log_error=/tmp/tmp.VU380MxLV9
+ log_generic daemon.notice WSREP: Running position recovery with --log_error=/tmp/tmp.VU380MxLV9
+ priority=daemon.notice
+ shift
+ date +%y%m%d %H:%M:%S
+ msg=130418 19:24:35 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.VU380MxLV9
+ echo 130418 19:24:35 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.VU380MxLV9
130418 19:24:35 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.VU380MxLV9
+ echo 130418 19:24:35 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.VU380MxLV9
+ LD_PRELOAD=/usr/lib/libjemalloc.so.1 nohup /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/lib/mysql/Buu.err --open-files-limit=65535 --pid-file=/var/lib/mysql/Buu.pid --log_error=/tmp/tmp.VU380MxLV9 --wsrep-recover
/usr/bin/mysqld_safe: 217: /usr/bin/mysqld_safe: LD_PRELOAD=/usr/lib/libjemalloc.so.1: not found
+ grep WSREP: Recovered position: /tmp/tmp.VU380MxLV9
+ rp=
+ [ -z ]
+ grep WSREP /tmp/tmp.VU380MxLV9
+ grep skipping position recovery
+ skipped=
+ [ -z ]
+ cat /tmp/tmp.VU380MxLV9
+ log_error WSREP: Failed to recover position:
+ log_generic daemon.error WSREP: Failed to recover position:
+ priority=daemon.error

=========================================================================

It failed there silently.

Full log for 5.5.29 here http://sprunge.us/ZRTO

Alex Yurchenko (ayurchen) wrote :

Fix pushed in lp:codership-mysql/5.5-23 r3864

Changed in codership-mysql:
status: Confirmed → Fix Committed
Frederic Descamps (lefred) wrote :

Raghu: it wasn't able to perform wsrep-recover but mysqld was able to run

Changed in percona-xtradb-cluster:
status: Fix Committed → Fix Released
Changed in percona-xtradb-cluster:
milestone: none → 5.5.30-23.7.4-2

@Fred,

Yes, that was also a bug. It should fail if it was not able to
wsrep-recover, than fail silently and proceed.

Changed in codership-mysql:
milestone: 5.5.30-24.8 → 5.5.31-23.7.4
Changed in codership-mysql:
status: Fix Committed → Fix Released

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1338

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers