XtraDB cluster does not properly detect bonded network interfaces

[Swany]

when using bonded network interfaces:
WSREP: Failed to read output of: '/sbin/ifconfig | grep -m1 -1 -E '^[a-z]?eth[0-9]' | tail -n 1 | awk '{ print $2 }' | awk -F : '{ print $2 }''
[Warning] WSREP: Failed to autoguess base node address
[Note] WSREP: Service disconnected.
[Note] WSREP: Some threads may fail to exit.

This might work better and should be interface type agnostic:
ping -i 1 -c 1 $(hostname) | grep -m1 -1 -E '\([0-9.]+{4}\)' | awk '{ print $3 }' | sed 's/[()]//g' | grep -v '127.0.0.1'

There should also probably be an option to set the preferred IP via an environment variable before installing the RPM. It may be the case that multiple interfaces exist, and the administrator prefers to use a particular interface (not just the first one from the list, or the one associated with the hostname). For unattended installations, setting the interface automatically may not work.

[Maciej Dobrzanski (mushu)]

I think pinging $(hostname) is not a good solution either, because it also relies on something that does not have to exist (e.g. hostname may not resolve into anything). The following command finds all non-local IP addresses associated with network interfaces.

/sbin/ifconfig | sed -e '/./{H;$!d};x;/LOOPBACK/d;/inet addr:.\+ UP [ A-Z]\+RUNNING/!d;s/.*inet addr:\([0-9.]\+\).*/\1/'

'| head -1' can be appended to automatically choose the first IP address. However as ifconfig is begin deprecated, it might be a good idea to use /sbin/ip instead.

[Baron Schwartz (baron-xaprb)]

The sed command suggested in the last comment may not be completely
reliable and portable. I've found that sed works differently in some
cases with respect to regex patterns and so forth. The command doesn't
work on my laptop, for example.

Perl might actually be a better tool for this case:

/sbin/ifconfig | perl -e
'$/="";while(<>){$_!~m/LOOPBACK/&&/UP.*RUNNING/&&s/^.*addr:(\S+).*$/$1/s&&print
$_, "\n"}'

[Maciej Dobrzanski (mushu)]

Perl is no longer shipped with the base system in some Linux distributions, so I guess Python would be a better choice these days. But this should be done with simple tools if portability is important. I don't see any reason why not to replace the one-liner with a call to an external bash script that would have a lot more flexibility in handling the task.

[Baron Schwartz (baron-xaprb)]

I agree on a script instead of a one-liner.

[Alex Yurchenko (ayurchen)]

I can't confirm this with the current mysql-wsrep code. In fact, no wsrep initialisation, and therefore address guessing should happen during RPM installation. If I understand it correctly, the only reason to run mysqld during installation is to create the base data directory and this can (and should) be done in a standalone mode.

[Piotr Biel (piotr-biel)]

Alex,

This is a bigger problem. It affects not only installation/initialization but it affects entire cluster.

With:
wsrep_sst_receive_address=bond0 10.1.4.20

It can't use xtrabackup/rsync for SST:

[ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup 'donor' 'bond0 10.1.2.10:4444/xtrabackup_sst'

nc: port range not valid

This is because:

+ nc bond0 10.1.4.21 4444

is being executed.

[Piotr Biel (piotr-biel)]

120604 3:10:08 [Warning] WSREP: Gap in state sequence. Need state transfer.
120604 3:10:10 [ERROR] WSREP: Failed to read output of: '/sbin/ifconfig | grep -m1 -1 -E '^[a-z]?eth[0-9]' | tail -n 1 | awk '{ print $2 }' | awk -F : '{ print $2 }''
120604 3:10:10 [ERROR] WSREP: Could not prepare state transfer request: failed to guess address to accept state transfer at. wsrep_sst_receive_address must be set manually.
120604 3:10:10 [ERROR] Aborting

Now, if I set:

wsrep_sst_receive_address=bond0 10.1.4.20

it ends up with issue described above.

[Piotr Biel (piotr-biel)]

With:

wsrep_sst_receive_address=10.1.4.20

works fine.

[Vadim Tkachenko (vadim-tk)]

Alex,

The call /sbin/ifconfig happens from mysqld.
And the main point is to use external script, not hardcoded line from mysqld.

[Alex Yurchenko (ayurchen)]

Vadim,

I know that this ifconfig call is invoked from mysqld. What I meant is that if it is happening _during_ install, the bug is not even that it fails to detect the right address, but that this call happens at all. And I suspect that this bug is either in Percona's mysqld code or in RPM installation script since I could not reproduce that with our rpms.

As for the script - are you seriously considering to write a script that would 100% correctly autodetect the right IP address on the right interface in the installations targeted for HA and advise people to rely on that instead of the manual configuration?

[Alex Yurchenko (ayurchen)]

Here's another manifestation of that bug: http://forum.percona.com/index.php?t=rview&goto=8749#msg_8749
- again wsrep provider (Galera) is loaded during installation.

[Launchpad Janitor (janitor)]

[Expired for Percona XtraDB Cluster because there has been no activity for 60 days.]

[churnd (churnd)]

I'm having this exact problem as well when trying to use xtrabackup for SST. Was there ever a recommended solution?

[churnd (churnd)]

Please disregard the previous comment. My issue was related to selinux being enabled, not this particular bug.

[Raghavendra D Prabhu (raghavendra-prabhu)]

@churnd,

Setting wsrep_node_address should do.

[Launchpad Janitor (janitor)]

[Expired for Percona XtraDB Cluster because there has been no activity for 60 days.]

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1219