Using xtrabackup sst fails due to mktemp error

[MentalPower]

Whenever a joining cluster node tries to join a cluster and its setup to do so via SST it will fail due to an issue when calling the mktemp helper program.

Older versions of mktemp allowed trailing characters after the string of XXXXXX for the template. Version 7.4 does not and exits with an error of "mktemp: too few X's in template `wsrep_sst_xtrabackupXXXX.pid'" if you try. Newer versions have a --suffix option to address this, but that is not an option on a default ubuntu 10.04 system.

We're fixing the issue manually by changing line 118 in wsrep_sst_xtrabackup from "XTRABACKUP_PID=$(mktemp --tmpdir wsrep_sst_xtrabackupXXXX.pid)" to just "XTRABACKUP_PID=$(mktemp --tmpdir wsrep_sst_xtrabackupXXXX)"

[Raghavendra D Prabhu (raghavendra-prabhu)]

It has been fixed in later versions of mktemp as you mentioned:

--suffix=SUFF
               append SUFF to TEMPLATE. SUFF must not contain slash. This option is implied if TEMPLATE does not end in X.

However, XTRABACKUP_PID, as the comment states, does nothing
there. It needs to be removed. Also, an alternative way of
check_pid needs to be used.

[Miguel Angel Nieto (miguelangelnieto)]

Bug confirmed. This is the output on Debian Squeeze:

# mktemp --version
mktemp (GNU coreutils) 8.5
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jim Meyering and Eric Blake.

# mktemp --tmpdir wsrep_sst_xtrabackupXXXX.pid
/tmp/wsrep_sst_xtrabackupAl4I.pid

This is the output on Ubuntu 10.04

# mktemp --version
mktemp (GNU coreutils) 7.4
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jim Meyering.

# mktemp --tmpdir wsrep_sst_xtrabackupXXXX.pid
mktemp: too few X's in template `wsrep_sst_xtrabackupXXXX.pid'

[Raghavendra D Prabhu (raghavendra-prabhu)]

Reported the XTRABACKUP_PID bug as lp:1086280 (and this bug should be marked as duplicate of that)

[Alex Yurchenko (ayurchen)]

Turns out, innobackupex implicitly stores its PID into predefined location: <tmpdir>/xtrabackup_pid. So using mktemp to create a unique pid file name is totally pointless and breaks PID checking functionality. Reverted to using ${TMPDIR}/xtrabackup_pid.

Pushed in lp:codership-mysql/5.5 r3830

[Raghavendra D Prabhu (raghavendra-prabhu)]

>Turns out, innobackupex implicitly stores its PID into
>predefined location: <tmpdir>/xtrabackup_pid. So using mktemp to
>create a unique pid file name is totally pointless and breaks
>PID checking functionality. Reverted to using
>${TMPDIR}/xtrabackup_pid.

Actually it doesn't store its PID, it stores the pid of
xtrabackup which it forks. So, by killing/sending signal to
xtrabackup process, the innobackupex may / may not be affected (depends on the location in code where the signal was delivered, hence non-deterministic). Hence I raised the request lp:1086286 . Till then, it is better to stick to pgrep / pidof if required.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Actually, while having the same fix as in https://bazaar.launchpad.net/~codership/codership-mysql/wsrep-5.5/revision/3830 it is possible to do what is mentioned in #5.

Just the kill_xtrabackup function requires change:
 https://bazaar.launchpad.net/~codership/codership-mysql/wsrep-5.5/view/3830/scripts/wsrep_sst_xtrabackup.sh#L41

ie. in
===========
kill_xtrabackup()
{
#set -x
    local PID=$(cat $XTRABACKUP_PID)
    [ -n "$PID" -a "0" != "$PID" ] && kill $PID && (kill $PID && kill -9 $PID) || :
    rm -f "$XTRABACKUP_PID"
#set +x
}
===========

Instead of killing the PID, kill the PPID (obtained with grep PPid /proc/$PID/status | cut -d: -f2), that should kill the innobackupex itself.

(Also, in kill $PID && (kill $PID && kill -9 $PID) --> shouldn't it be (kill $PID || (kill $PID || kill -9 $PID) ?)

[Alex Yurchenko (ayurchen)]

So which process do we actually want to kill:
- xtrabackup?
- innobackupex?
- both?
(I'm somewhat confused by this innobackupex/xtrabackup duality... why do we have 2 of them?)

[Vadim Tkachenko (vadim-tk)]

Alex,

You can read a difference there
http://www.percona.com/doc/percona-xtrabackup/manual.html

We need to kill both of them if a caller died.

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Alex,

I tested and yes, you need to kill both.

[nntp (smartjoe)]

Alex,

We have same problem when deploy on SuSE Enterprise Linux 11 SP2 (x86_64). (Install from source)

But it is strange the problem doesn't happen when deploy the same thing inside virtualbox with the same Linux OS and patch level.

Joe

[nntp (smartjoe)]

Sorry missed the mktemp version in SuSE Enterprise Linux 11 SP2

mktemp 8.12

[Alex Yurchenko (ayurchen)]

Joe, please try an updated wsrep_sst_xtrabackup - https://launchpad.net/codership-mysql/5.5/5.5.28-23.7/+download/wsrep_sst_xtrabackup

The very idea of using mktmp turned out to be wrong.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1257