Percona XtraDB Cluster - HA scalable solution for MySQL

nmap causes MySQL crash

Reported by Kate Cronin on 2013-05-24
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Galera
High
Teemu Ollakka
Percona XtraDB Cluster
Undecided
Unassigned

Bug Description

Basic info:

# uname -a
Linux 2.6.32-235.el6.x86_64 #1 SMP Fri Feb 17 11:48:53 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa | grep -i percona
percona-xtrabackup-2.0.7-552.rhel6.x86_64
Percona-XtraDB-Cluster-shared-5.5.30-23.7.4.406.rhel6.x86_64
percona-toolkit-2.1.4-1.noarch
Percona-XtraDB-Cluster-client-5.5.30-23.7.4.406.rhel6.x86_64
Percona-Server-shared-compat-5.5.30-rel30.2.508.rhel6.x86_64
Percona-XtraDB-Cluster-server-5.5.30-23.7.4.406.rhel6.x86_64
Percona-XtraDB-Cluster-galera-2.5-1.150.rhel6.x86_64

I can quite reliably crash MySQL by running 'nmap' against the server running it, but only as a non-root user. If I run nmap as root, all's well. I think the difference lies in how nmap scans ports when run as root vs. non-root. Some strace output:

as root, it uses "sendto()":
sendto(3, "E\0\0,\260x\0\0(\6\211#\212H\204\320\212H\277\317\364v\21\327\261\"?\313\0\0\0\0"..., 44, 0, {sa_family=AF_INET, sin_port=htons(4567), sin_addr=inet_addr("xxx.xxx.xxx.xxx")}, 16) = 44

as non-root, it uses "connect()":
connect(3, {sa_family=AF_INET, sin_port=htons(4567), sin_addr=inet_addr("xxx.xxx.xxx.xxx")}, 16) = -1 EINPROGRESS (Operation now in progress)

(I don't know that it's the connection to port 4567 that is the culprit; just used those lines from strace output as examples).

MySQL dies like so:

terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<asio::system_error> >'
  what(): Transport endpoint is not connected
22:47:10 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=11
max_threads=2002
thread_count=9
connection_count=9
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4389869 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7d0765]
/usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x6a9c64]
/lib64/libpthread.so.0(+0xf520)[0x7f0ddfe05520]
/lib64/libc.so.6(gsignal+0x35)[0x7f0dde9dba45]
/lib64/libc.so.6(abort+0x175)[0x7f0dde9dd225]
/usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x12d)[0x7f0ddc13ca7d]
/usr/lib64/libstdc++.so.6(+0xbcc06)[0x7f0ddc13ac06]
/usr/lib64/libstdc++.so.6(+0xbcc33)[0x7f0ddc13ac33]
/usr/lib64/libstdc++.so.6(+0xbcd2e)[0x7f0ddc13ad2e]
/usr/lib64/libgalera_smm.so(_ZN5boost15throw_exceptionIN4asio12system_errorEEEvRKT_+0x18a)[0x7f0ddc4a93ca]
/usr/lib64/libgalera_smm.so(_ZN4asio6detail11throw_errorERKNS_10error_codeE+0x5b)[0x7f0ddc4a94eb]
/usr/lib64/libgalera_smm.so(_ZNK4asio12basic_socketINS_2ip3tcpENS_21stream_socket_serviceIS2_EEE15remote_endpointEv+0xc1)[0x7f0ddc4a9671]
/usr/lib64/libgalera_smm.so(_ZN5gcomm13AsioTcpSocket18assign_remote_addrEv+0x5e7)[0x7f0ddc49a657]
/usr/lib64/libgalera_smm.so(_ZN5gcomm15AsioTcpAcceptor14accept_handlerEN5boost10shared_ptrINS_6SocketEEERKN4asio10error_codeE+0xc4)[0x7f0ddc49b0e4]
/usr/lib64/libgalera_smm.so(_ZN4asio6detail25reactive_socket_accept_opINS_12basic_socketINS_2ip3tcpENS_21stream_socket_serviceIS4_EEEES4_N5boost3_bi6bind_tIvNS8_4_mfi3mf2IvN5gcomm15AsioTcpAcceptorENS8_10shared_ptrINSD_6SocketEEERKNS_10error_codeEEENS9_5list3INS9_5valueIPSE_EENSN_ISH_EEPFNS8_3argILi1EEEvEEEEEE11do_completeEPNS0_15task_io_serviceEPNS0_25task_io_service_operationESI_m+0x180)[0x7f0ddc4a4eb0]
/usr/lib64/libgalera_smm.so(_ZN4asio6detail15task_io_service3runERNS_10error_codeE+0x459)[0x7f0ddc4c68c9]
/usr/lib64/libgalera_smm.so(_ZN5gcomm12AsioProtonet10event_loopERKN2gu8datetime6PeriodE+0x1d6)[0x7f0ddc4be816]
/usr/lib64/libgalera_smm.so(_ZN9GCommConn3runEv+0x57)[0x7f0ddc4d8777]
/usr/lib64/libgalera_smm.so(_ZN9GCommConn6run_fnEPv+0x9)[0x7f0ddc4dd469]
/lib64/libpthread.so.0(+0x77e1)[0x7f0ddfdfd7e1]
/lib64/libc.so.6(clone+0x6d)[0x7f0ddea8f8ed]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
130524 15:47:10 mysqld_safe Number of processes running now: 0
130524 15:47:10 mysqld_safe WSREP: not restarting wsrep node automatically
130524 15:47:10 mysqld_safe mysqld from pid file /mnt/ssd/mysql/xxx.pid ended

I saw this discussion of a very similar problem on the "Percona discussions" google group, but I'm not sure an actual bug was ever submitted - a search on this site didn't find anything, but I apologize if this ends up being a duplicate.

https://groups.google.com/forum/?fromgroups#!topic/percona-discussion/gUWvXOKR_88

Thanks!

description: updated
affects: percona-server → percona-xtradb-cluster

Seems to be related to https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1153727 .. you can follow updates in both places.

I can reproduce this.

However, we strongly recommend that cluster ports be secured externally as well with iptables/pf since otherwise other ports of cluster are vulnerable as well - like anyone will be able to SST from a donor since no checks are performed there against any whitelist/blacklist.

But, again, this shouldn't crash the server. It is being investigated in lp:1153727.

Changed in percona-xtradb-cluster:
status: New → Confirmed

I can also reproduce that with/without root the behaviour is different wrt. crash.

Laurent Minost (lolomin) wrote :

Hi Raghavendra,

Thanks for these informations !
I've faced the same crash on one of my development PXDBC node and the interrogation is : what can be done exactly to avoid this bug ? Firewalling of ports 4444 and 4567 from IP others than the one used by the cluster nodes should be enough ?
To better understand the problem what will be done on your side to fix this, what is the root cause of this sudden crash ?

Regards,

Laurent

Changed in galera:
assignee: nobody → Teemu Ollakka (teemu-ollakka)
importance: Undecided → High
milestone: none → 23.2.6
status: New → Confirmed
Changed in percona-xtradb-cluster:
milestone: none → 5.5.31-24.8
Changed in galera:
status: Confirmed → In Progress
Laurent Minost (lolomin) wrote :

Hi,

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-05-25: #2
"I can reproduce this.

However, we strongly recommend that cluster ports be secured externally as well with iptables/pf since otherwise other ports of cluster are vulnerable as well - like anyone will be able to SST from a donor since no checks are performed there against any whitelist/blacklist."

Could you please precise this behavior ? Is it really possible presently to get a complete SST from an outside attacker which gained access to the LAN on which a cluster is located when this cluster uses xtrabackup as SST method ? I mean : there is no authentication (like wsrep_sst_auth) used between joiner and donor ?

Thanks !

Regards,

Laurent

Alex Yurchenko (ayurchen) wrote :

Laurent,

1) It depends on your SST script. E.g. mysqldump uses authentication, rsync does not. So of course if rsync daemon is running on your node, and it is accessible from the insecure network, then indeed, an attacker can get a snapshot of your data directory.

2) For authentication to really be of any protection, it (and data transfer) must be performed over a well-encrypted channel. So speaking about authentication without encryption is moot. And so far in the encryption department people trust VPN or SSH/SSL tunnels. So you know what to do - if you are in insecure network.

Regards,
Alex

Alex Yurchenko (ayurchen) wrote :

fix committed in r152

Changed in galera:
status: In Progress → Fix Committed
Changed in percona-xtradb-cluster:
status: Confirmed → Fix Released
Changed in galera:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers