SST on IPv6 fails for xtrabackup-v2

[Mario Splivalo]

I have three nodes configured on a ipv4+ipv6 mixed network. Both of my nodes have ipv4 and ipv6 addresses, however I have configured percona to use only ipv6. I am able to connect to 3306 port from/to each of the machines using only ipv6.

Due to issues with having IPv6 addresses in the configuration file I added the hostnames of the participating nodes in /etc/hosts file, with accompanying IPv6 addresses so that I'm sure that all the traffic goes over IPv6.

However, joiners can't receive data from donor because 'socat' on joiner is only listeninig on ipv4:

mysql 20553 20394 0 18:17 pts/1 00:00:00 socat -u TCP-LISTEN:4444,reuseaddr stdio

If, using the hostname, I try to connect, from donnor node to joiner node, I can't connect:

root@machine-1:/var/lib/mysql# telnet machine-2 4444
Trying 2001:db8:1:0:f816:3eff:fea4:2dd8...
telnet: Unable to connect to remote host: Connection refused

When I use ipv4 address, all is good:

root@machine-1:/var/lib/mysql# host machine-2
machine-2.openstacklocal has address 10.5.19.139
root@machine-1:/var/lib/mysql# telnet 10.5.19.139 4444
Trying 10.5.19.139...
Connected to 10.5.19.139.
Escape character is '^]'.

Of course, when I force socat to use TCP6-LISTEN (by hacking into the /usr/bin/wsrep_sst_xtrabackup-v2) SST finishes fine.

(As a workaround, when using rsync as a wsrep_sst_method, I am able to complete the SST, regardless of the issue mentioned in bug #1315528).

[Mario Splivalo (mariosplivalo)]

As a better workaround I created a new SST plugin - called it xtrabackup-v2-ipv6. I merely copied /usr/bin/wsrep_sst_xtrabackup-v2 to /usr/bin/wsrep_sst_xtrabackup-v2-ipv6, and altered it so that socat always uses TCP6-LISTEN. When that new plugin is deployed across all the cluster members I have SST using xtrabackup over IPv6 working.

[Xiang Hui (xianghui)]

The TCP-LISTEN is not used only for ipv4, it could be set with export "SOCAT_DEFAULT_LISTEN_IP={4,6}" for socat to listening on different ip protocol, so no need to create a new wsrep_sst_xtrabackup-v2-ipv6 plugin.

xianghui@Thinkpad-x240:~/workplace/github$ export SOCAT_DEFAULT_LISTEN_IP=6
xianghui@Thinkpad-x240:~/workplace/github$ socat -u TCP-LISTEN:4444,reuseaddr stdio

xianghui@Thinkpad-x240:~/workplace/github$ telnet :: 4444
Trying ::...
Connected to ::.
Escape character is '^]'.

[Mario Splivalo (mariosplivalo)]

Actually, imho, the best fix for this would be for the percona-cluster init script to honor /etc/default - one could put all the required stuff inside /etc/default/percona thus eliminating the need of fiddling with init scripts or provided wsrep plugins.

[Mario Splivalo (mariosplivalo)]

I opened bug #1381492, regarding /etc/default issue.

[Mario Splivalo (mariosplivalo)]

And, here is even better fix - one just has to read documentation thoroughly :)

All the wsrep_* plugins should honor the [sst] section inside my.cnf configuration file (as per this: http://www.percona.com/doc/percona-xtradb-cluster/5.5/manual/xtrabackup_sst.html#sockopt)

So, this issue is easily fixed by adding this snippet to my.cnf:

[sst]
sockopt=,pf=ip6

[Miguel Angel Nieto (miguelangelnieto)]

You are right Mario, that's the correct way to change IP protocol for SST. This is not a bug but I still think it worth adding your example to the documentation. It will help people to understand how to use that parameter and also help the users using ipv6 (the adoption is increasing).

I'm going to confirm the bug, but only as a documentation bug in order to improve it a bit more.

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Mario, @Xiang,

Yes, /etc/default/mysql should be honored on debian now, so you can do like SOCAT_DEFAULT_LISTEN_IP=6 in that file.

[Mario Splivalo (mariosplivalo)]

To clear some possible confusion - this bag is, actually, invalid. When I first opened it i wasn't aware of [sst] option in the config file for percona-xtradb-cluster.

So, to make xtrabackup-v2 plugin actually use ipv6 when doing sst, this has to be added to my.cnf:

[sst]
sockopt=,pf=ip6

So, this is actually a documentation issue, as tagged by Miguel.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1753