pt-table-sync render "Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client is deprecated" message

[NJ Johansson]

Running "pt-table-sync --print SOURCEDSN TARGETDSN" on a default RHEL 7 install renders this message:

*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 together with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at /usr/bin/pt-table-sync line 8607.

This is what's on line 8607 and onwards:

       if ( $scheme eq 'https') {
           IO::Socket::SSL->start_SSL($self->{fh});
           ref($self->{fh}) eq 'IO::Socket::SSL'
               or die(qq/SSL connection failed for $host\n/);
           if ( $self->{fh}->can("verify_hostname") ) {
               $self->{fh}->verify_hostname( $host, $ssl_verify_args )
                  or die(qq/SSL certificate not valid for $host\n/);
           }
           else {
            my $fh = $self->{fh};
            _verify_hostname_of_cert($host, _peer_certificate($fh), $ssl_verify_args)
                  or die(qq/SSL certificate not valid for $host\n/);
            }
       }

$ pt-table-sync --version
pt-table-sync 2.2.17

$ cat /etc/*release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.2 (Maipo)"

$ rpm -qi mariadb101u-server
Name : mariadb101u-server
Epoch : 1
Version : 10.1.14
Release : 2.ius.el7

https://www.percona.com/forums/questions-discussions/percona-toolkit/20917-pt-table-sync-causes-ssl_verify_mode-error-due-to-latest-io-socket-ssl refers to http://stackoverflow.com/questions/15884630/selecting-ssl-verify-none-for-ssl-verify-mode that says it's a new default in Perl package IO::Socket::SSL

For reference I also noticed this issue being reported for XtraBackup (https://bugs.launchpad.net/percona-xtrabackup/+bug/1557991)

[NJ Johansson (nj.johansson)]

Suggested fix which seems to alleviate this problem for me:

$ pt-table-sync --version
pt-table-sync 2.2.18

$ diff /usr/bin/pt-table-sync.bak160716 /usr/bin/pt-table-sync
8629c8629,8631
< IO::Socket::SSL->start_SSL($self->{fh});
---
> # 0x01 corresponds to SSL_VERIFY_PEER
> IO::Socket::SSL->start_SSL($self->{fh}, SSL_verify_mode => 0x01);

[NJ Johansson (nj.johansson)]

This is still an issue after upgrading to 2.2.20 but the same fix as suggested above works:

# pt-table-sync --version
pt-table-sync 2.2.20

# diff /usr/bin/pt-table-sync.bak161212 /usr/bin/pt-table-sync
8647c8647,8648
< IO::Socket::SSL->start_SSL($self->{fh});
---
> # 0x01 corresponds to SSL_VERIFY_PEER
> IO::Socket::SSL->start_SSL($self->{fh}, SSL_verify_mode => 0x01);

[NJ Johansson (nj.johansson)]

Just upgraded to 3.0.1 and can confirm this is still an issue.

$ pt-table-sync --version
pt-table-sync 3.0.1

Fixed it as usual using:

$ diff /usr/bin/pt-table-sync.bak170222 /usr/bin/pt-table-sync
8655c8655,8656
< IO::Socket::SSL->start_SSL($self->{fh});
---
> # 0x01 corresponds to SSL_VERIFY_PEER
> IO::Socket::SSL->start_SSL($self->{fh}, SSL_verify_mode => 0x01);

[Jericho Rivera (jericho-rivera)]

Can you please share us the full pt-table-sync command you are using. Also would you be able to run pt-table-sync with PTDEBUG=1 and share the full output here please.

[NJ Johansson (nj.johansson)]

As I stated originally, pt-table-sync --print SOURCEDSN TARGETDSN has been enough for me to trigger it.
We have a command running every minute in crontab that also triggers it, which is (credentials and specifics redacted):
pt-table-sync --execute --no-check-slave --databases xx --tables xx,yy,zz \
  h=mysql-1.xx,P=3306,u=xx,p=yy \
  h=mysql-2.yy,P=3306,u=xx,p=yy

The interesting thing is that I've noticed IO::Socket::SSL doesn't spit this error out every time like if it's cached somewhere? Not sure if there is any reuse of connections going on either in your package or globally for IO::Socket::SSL?
I just tried triggered it but it didn't spit this error out (so I couldn't give you PTDEBUG=1 output) but I know for sure it will after some time so to replicate it I would advice you to create a small script running every minute until error pops up.
Judging by how many other people I can find on the internet with the same error makes me confident I'm not alone.

Also, the underlying reason why this error is triggered should be something you should think about. Should SSL certificates be verified by default or not? If it's your intention not to, maybe it's better to be explicit about it in your code?

I tried checking IO::Socket::SSL source for version 1.94 which is on our server (http://cpansearch.perl.org/src/SULLR/IO-Socket-SSL-1.94/lib/IO/Socket/SSL.pm). I'm not using SSL connections to any of the hosts so one interesting aspect is also why pt-table-sync gets inside this code at all? (where it is obviously triggered)
if ( $scheme eq 'https') {

This should probably give you a good hint about what's going on here and some more background to research it properly!

[Jericho Rivera (jericho-rivera)]

I believe the SSL check happens when version checking is triggered.

# VersionCheck:9184 608 3 instances to check
# VersionCheck:9194 608 Using https
*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 together with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at /bin/pt-table-sync line 8655.
# VersionCheck:9360 608 Server response: $VAR1 = {
# content => 'OS;os_version
# MySQL;mysql_variable;version_comment,version
# Perl;perl_version
# DBD::mysql;perl_module_version
# Percona::Toolkit;perl_module_version',
# headers => {
# 'cache-control' => 'no-cache',
# connection => 'close',
# 'content-length' => '144',
# 'content-type' => 'text/plain; charset=UTF-8',
# date => 'Tue, 21 Mar 2017 07:34:40 GMT',
# server => 'nginx',
# 'set-cookie' => 'PHPSESSID=svqea1ergjk81udq5uhge4kt16; path=/',
# 'strict-transport-security' => 'max-age=31536000; includeSubdomains;',
# vary => 'Accept-Encoding',
# 'x-content-type-options' => 'nosniff',
# 'x-frame-options' => 'SAMEORIGIN',
# 'x-powered-by' => 'Percona XtraDB',
# 'x-xss-protection' => '1; mode=block'
# },
# protocol => 'HTTP/1.1',
# reason => 'OK',
# status => '200',
# success => 1
# };
#
# VersionCheck:9461 608 Items: $VAR1 = {
# DBD::mysql => {
# item => 'DBD::mysql',
# type => 'perl_module_version',
# vars => []
# },
# MySQL => {
# item => 'MySQL',
# type => 'mysql_variable',
# vars => [
# 'version_comment',
# 'version'
# ]
# },
# OS => {
# item => 'OS',
# type => 'os_version',
# vars => []
# },
# Percona::Toolkit => {
# item => 'Percona::Toolkit',
# type => 'perl_module_version',
# vars => []
# },
# Perl => {
# item => 'Perl',
# type => 'perl_version',
# vars => []
# }
# };
#
# VersionCheck:9605 608 Perl version for $Percona::Toolkit::VERSION = 3.0.0
# VersionCheck:9605 608 Perl version for $DBD::mysql::VERSION = 4.023
# VersionCheck:9594 608 Perl version 5.16.3
# VersionCheck:9520 608 platform: Linux
# VersionCheck:9525 608 lsb_release:
# VersionCheck:9584 608 OS version = CentOS Linux release 7.3.1611 (Core)
# VersionCheck:9641 608 SHOW VARIABLES
# VersionCheck:9648 608 MySQL version for MySQL = Percona Server (GPL), Release 80.0, Revision f113994f31 on c6a3306
# VersionCheck:9648 608 MySQL version for MySQL = 5.6.35-80.0-log on c6a3306
# VersionCheck:9641 608 SHOW VARIABLES
# VersionCheck:9648 608 MySQL version for MySQL = Percona Server (GPL), Release 80.0, Revision f113994f31 on c6b3306
# VersionCheck:9648 608 MySQL version for MySQL = 5.6.35-80.0-log on c6b3306
# VersionCheck:9391 608 Client response: $VAR1 = {
# content => '4a706353fd354a93fed09bfed745629e;DBD::mysql;4.023
# 0c1fef87a903d37664666714b3c5a45d;MySQL;Percona Server (GPL),...

[NJ Johansson (nj.johansson)]

Ah, now it makes sense!
I was a bit confused over why I only received this error message exactly once per day.
In the source at https://www.percona.com/get/pt-table-sync I can see the version check is calling this function:

sub version_check_time_limit {
   return 60 * 60 * 24; # one day
}

I.e. version check is only triggered once per day = error message is only output once daily. Puzzle solved!

Given this is a secure connection terminating at a server you control (https://v.percona.com) I would strongly advise you to make sure SSL_VERIFY_PEER is set (per my earlier suggested patch code) to avoid the risk of man-in-the-middle attacks.

[Jericho Rivera (jericho-rivera)]

The workaround would be to use --no-version-check but indeed using SSL_verify_mode=SSL_VERIFY_PEER would be the fix.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PT-1363