Percona XtraDB Cluster - HA scalable solution for MySQL

Using xtrabackup sst fails due to mktemp error

Reported by MentalPower on 2012-11-19
28
This bug affects 5 people
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
Critical
Alex Yurchenko
Percona XtraDB Cluster
Undecided
Unassigned

Bug Description

Whenever a joining cluster node tries to join a cluster and its setup to do so via SST it will fail due to an issue when calling the mktemp helper program.

Older versions of mktemp allowed trailing characters after the string of XXXXXX for the template. Version 7.4 does not and exits with an error of "mktemp: too few X's in template `wsrep_sst_xtrabackupXXXX.pid'" if you try. Newer versions have a --suffix option to address this, but that is not an option on a default ubuntu 10.04 system.

We're fixing the issue manually by changing line 118 in wsrep_sst_xtrabackup from "XTRABACKUP_PID=$(mktemp --tmpdir wsrep_sst_xtrabackupXXXX.pid)" to just "XTRABACKUP_PID=$(mktemp --tmpdir wsrep_sst_xtrabackupXXXX)"

Related branches

It has been fixed in later versions of mktemp as you mentioned:

--suffix=SUFF
               append SUFF to TEMPLATE. SUFF must not contain slash. This option is implied if TEMPLATE does not end in X.

However, XTRABACKUP_PID, as the comment states, does nothing
there. It needs to be removed. Also, an alternative way of
check_pid needs to be used.

Bug confirmed. This is the output on Debian Squeeze:

# mktemp --version
mktemp (GNU coreutils) 8.5
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jim Meyering and Eric Blake.

# mktemp --tmpdir wsrep_sst_xtrabackupXXXX.pid
/tmp/wsrep_sst_xtrabackupAl4I.pid

This is the output on Ubuntu 10.04

# mktemp --version
mktemp (GNU coreutils) 7.4
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jim Meyering.

# mktemp --tmpdir wsrep_sst_xtrabackupXXXX.pid
mktemp: too few X's in template `wsrep_sst_xtrabackupXXXX.pid'

Changed in percona-xtradb-cluster:
status: New → Confirmed

Reported the XTRABACKUP_PID bug as lp:1086280 (and this bug should be marked as duplicate of that)

Changed in codership-mysql:
milestone: none → 5.5.28-23.8
status: New → Confirmed
Alex Yurchenko (ayurchen) wrote :

Turns out, innobackupex implicitly stores its PID into predefined location: <tmpdir>/xtrabackup_pid. So using mktemp to create a unique pid file name is totally pointless and breaks PID checking functionality. Reverted to using ${TMPDIR}/xtrabackup_pid.

Pushed in lp:codership-mysql/5.5 r3830

Changed in codership-mysql:
assignee: nobody → Alex Yurchenko (ayurchen)
status: Confirmed → Fix Committed
importance: Undecided → Critical

>Turns out, innobackupex implicitly stores its PID into
>predefined location: <tmpdir>/xtrabackup_pid. So using mktemp to
>create a unique pid file name is totally pointless and breaks
>PID checking functionality. Reverted to using
>${TMPDIR}/xtrabackup_pid.

Actually it doesn't store its PID, it stores the pid of
xtrabackup which it forks. So, by killing/sending signal to
xtrabackup process, the innobackupex may / may not be affected (depends on the location in code where the signal was delivered, hence non-deterministic). Hence I raised the request lp:1086286 . Till then, it is better to stick to pgrep / pidof if required.

Actually, while having the same fix as in https://bazaar.launchpad.net/~codership/codership-mysql/wsrep-5.5/revision/3830 it is possible to do what is mentioned in #5.

Just the kill_xtrabackup function requires change:
 https://bazaar.launchpad.net/~codership/codership-mysql/wsrep-5.5/view/3830/scripts/wsrep_sst_xtrabackup.sh#L41

ie. in
===========
kill_xtrabackup()
{
#set -x
    local PID=$(cat $XTRABACKUP_PID)
    [ -n "$PID" -a "0" != "$PID" ] && kill $PID && (kill $PID && kill -9 $PID) || :
    rm -f "$XTRABACKUP_PID"
#set +x
}
===========

Instead of killing the PID, kill the PPID (obtained with grep PPid /proc/$PID/status | cut -d: -f2), that should kill the innobackupex itself.

(Also, in kill $PID && (kill $PID && kill -9 $PID) --> shouldn't it be (kill $PID || (kill $PID || kill -9 $PID) ?)

Alex Yurchenko (ayurchen) wrote :

So which process do we actually want to kill:
- xtrabackup?
- innobackupex?
- both?
(I'm somewhat confused by this innobackupex/xtrabackup duality... why do we have 2 of them?)

Vadim Tkachenko (vadim-tk) wrote :

Alex,

You can read a difference there
http://www.percona.com/doc/percona-xtrabackup/manual.html

We need to kill both of them if a caller died.

@Alex,

I tested and yes, you need to kill both.

nntp (smartjoe) wrote :

Alex,

We have same problem when deploy on SuSE Enterprise Linux 11 SP2 (x86_64). (Install from source)

But it is strange the problem doesn't happen when deploy the same thing inside virtualbox with the same Linux OS and patch level.

Joe

nntp (smartjoe) wrote :

Sorry missed the mktemp version in SuSE Enterprise Linux 11 SP2

mktemp 8.12

Alex Yurchenko (ayurchen) wrote :

Joe, please try an updated wsrep_sst_xtrabackup - https://launchpad.net/codership-mysql/5.5/5.5.28-23.7/+download/wsrep_sst_xtrabackup

The very idea of using mktmp turned out to be wrong.

Changed in percona-xtradb-cluster:
milestone: none → 5.5.29-23.8
Changed in percona-xtradb-cluster:
milestone: 5.5.29-23.8 → 5.5.29-27.3.1
status: Confirmed → Fix Committed
milestone: 5.5.29-27.3.1 → 5.5.29-23.7.1
Changed in percona-xtradb-cluster:
status: Fix Committed → Fix Released
Changed in codership-mysql:
status: Fix Committed → Fix Released
Changed in codership-mysql:
milestone: 5.5.30-24.8 → 5.5.31-23.7.4
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers