Denial of Service: mysql-server going berserk when contacted by unauthorized client

[Hadmut Danisch]

Hi,

on one of our servers we noticed that under certain conditions mysql-server can be caused to go berserk, i.e. run with 400% CPU load, spit out extrem tons of log messages and denial it's work completely when contacted by a client, that is not (!) authorized to connect.

I've boiled this down to a simple scenario in a LXD box to demonstrate the problem:

1. create a fresh LXD container (or any environment of your choice with a fresh 16.04) and enter it:

lxc launch x mysqltest
lxc exec mysqltest -- /bin/bash

2. inside the container do

apt-get update
apt-get upgrade
apt-get install mysql-server

edit /etc/mysql/mysql.conf.d/mysqld.cnf and comment out the line bind-adress = 127.0.0.1

/etc/init.d/mysql restart

Verify with netstat or lsof -i that mysql is not bound to loopback but has a general socket open

Add the line
ALL: ALL
to /etc/hosts.deny

Add the line
mysqld: 1.2.3.4
to /etc/hosts.allow

3. contact the mysqld from outside from any host that is not (!) allowed to connect by /etc/hosts.*, e.g. just from your Host running LXD, with a simple

telnet 192.168.200.221 mysql
(use the ip address of your container, of course)

no need to enter anything

4. Inside the container: Watch the damon using all CPU (here: permanently oscillating between 100% and 400% CPU usage)

strace shows endless

[pid 4143] accept(-1, 0x7ffd120dbb20, 0x7ffd120dbabc) = -1 EBADF (Bad file descriptor)
[pid 4143] poll([{fd=28, events=POLLIN}, {fd=29, events=POLLIN}], 2, -1) = 1 ([{fd=28, revents=POLLNVAL}])

loop.

/var/log/mysql/error.log rapidly growing with lines like

2017-02-10T10:46:06.062292Z 0 [ERROR] Error in accept: Bad file descriptor

and thus filling the disk.

regards

[Seth Arnold (seth-arnold)]

Thanks for the bug Hadmut; I'm marking this public so that administrators can more quickly learn that using tcpwrappers for access control has the potential for trouble with mysqld, and can switch to iptables or other firewalling as appropriate.

[Robie Basak (racb)]

Hadmut, could you report the package version number of mysql-5.7 in which you are seeing this please? The command "dpkg-query -W mysql-server-5.7" will output this.

[Hadmut Danisch (hadmut)]

Robie,

I did not keep the virtual machine. On a host where the problem occured first we have

mysql-server-5.7 5.7.17-0ubuntu0.16.04.1

I just repeated the steps described above in a fresh lxc machine and again got

mysql-server-5.7 5.7.17-0ubuntu0.16.04.1

regards

[Daniel Black (daniel-black)]

upstream bug: https://bugs.mysql.com/bug.php?id=84708

"Usage of tcp wrappers is totally non-researched and not documented. Hence, our manual should contain recommendations on the usage of these daemons."

[Robie Basak (racb)]

Thank you for the update Daniel.

There's a further comment now from the same person in the upstream bug tracker: "Turns out that this is a real bug that needs fixing in the code"

So I'll mark this Triaged, but I don't expect that Ubuntu will be able to do anything about this until there is a fix released upstream.

[Jan Kellermann (jan-kellermann)]

The bug #84708 on mysql is fixed. Please have a look to fix this possible fast because all servers with recommended "ALL: PARANOID" in hosts.deny can be brought to their knees by a single connection from a misconfigurated ip-address (aka botnet).

[Jan Kellermann (jan-kellermann)]

Affected version:
# dpkg-query -W mysql-server-5.7
mysql-server-5.7 5.7.17-0ubuntu0.16.04.2

[Jan Kellermann (jan-kellermann)]

The bug is fixed in MySQL 5.7.19 - see https://forums.mysql.com/read.php?3,658909,658909

Link to mysql-bug: https://bugs.mysql.com/bug.php?id=84708

The Oracle-Bug-ID is 25476479

In Ubuntu-changelog are only Security-Errors marked:
https://launchpad.net/ubuntu/+source/mysql-5.7/5.7.19-0ubuntu1

Is this bug-fix also included?

best regards

[Andreas Hasenack (ahasenack)]

If it's indeed fixed in 5.7.19, then bionic and later are fixed. Can someone verify?

[Christian Ehrhardt  (paelzer)]

We had no further report, so we should (while cleaning up bugs) assume it indeed got fixed.
Since thereby >=Bionic is good and < Bionic has entered extended support I think this is done.