fail2ban fails to ban

[Alistair Buxton]

Some months ago I set up a server with fail2ban for ssh and wordpress. Initially it worked but since then it seems to have stopped working correctly and now nothing is ever banned.

The only modifed config files are filter.d/wordpress.filter and jail.local.

wordpress.filter:

# Fail2Ban configuration file
#
# Author: Charles Lecklider
#

[INCLUDES]

# Read common prefixes. If any customizations available -- read them from
# common.local
before = common.conf

[Definition]

_daemon = wordpress

# Option: failregex
# Notes.: regex to match the password failures messages in the logfile. The
# host must be matched by a group named "host". The tag "<HOST>" can
# be used for standard IP/hostname matching and is only an alias for
# (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
# Values: TEXT
#
failregex = ^%(__prefix_line)sAuthentication failure for .* from <HOST>$
            ^%(__prefix_line)sBlocked authentication attempt for .* from <HOST>$
            ^%(__prefix_line)sBlocked user enumeration attempt from <HOST>$
            ^%(__prefix_line)sPingback requested from <HOST>$

# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
#
ignoreregex =

jail.local:

[DEFAULT]

ignoreip = 127.0.0.0/8 83.104.131.111

[wordpress]
enabled = true
filter = wordpress
logpath = /var/log/auth.log
port = http
protocol = tcp
bantime = 3600
maxretry = 10

Running:

fail2ban-regex -v auth.log /etc/fail2ban/filter.d/sshd.conf

shows many hits from the same IP a few seconds apart, which should definitely have triggered a ban. For example:

| 82.165.145.148 Wed Dec 31 05:17:24 2014
| 82.165.145.148 Wed Dec 31 05:17:27 2014
| 82.165.145.148 Wed Dec 31 05:17:30 2014
| 82.165.145.148 Wed Dec 31 05:17:32 2014
| 82.165.145.148 Wed Dec 31 05:17:36 2014
| 82.165.145.148 Wed Dec 31 05:17:39 2014
| 82.165.145.148 Wed Dec 31 05:17:42 2014
| 82.165.145.148 Wed Dec 31 05:17:44 2014
| 82.165.145.148 Wed Dec 31 05:17:48 2014
| 82.165.145.148 Wed Dec 31 05:17:51 2014
| 82.165.145.148 Wed Dec 31 05:17:54 2014
| 82.165.145.148 Wed Dec 31 05:17:56 2014
| 87.106.151.114 Wed Dec 31 05:17:56 2014
| 82.165.145.148 Wed Dec 31 05:17:59 2014
| 87.106.151.114 Wed Dec 31 05:17:59 2014
| 87.106.151.114 Wed Dec 31 05:18:02 2014
| 87.106.151.114 Wed Dec 31 05:18:06 2014
| 87.106.151.114 Wed Dec 31 05:18:08 2014
| 87.106.151.114 Wed Dec 31 05:18:12 2014
| 87.106.151.114 Wed Dec 31 05:18:16 2014
| 87.106.151.114 Wed Dec 31 05:18:19 2014
| 87.106.151.114 Wed Dec 31 05:18:21 2014
| 82.165.145.148 Wed Dec 31 05:19:06 2014
| 82.165.145.148 Wed Dec 31 05:19:10 2014
| 82.165.145.148 Wed Dec 31 05:19:13 2014
| 82.165.145.148 Wed Dec 31 05:19:15 2014
| 82.165.145.148 Wed Dec 31 05:19:18 2014
| 82.165.145.148 Wed Dec 31 05:19:22 2014
| 82.165.145.148 Wed Dec 31 05:19:25 2014
| 82.165.145.148 Wed Dec 31 05:19:27 2014
| 82.165.145.148 Wed Dec 31 05:19:30 2014

However my /var/log/fail2ban.log contains only the following line:

2014-12-28 06:25:04,457 fail2ban.server : INFO Changed logging target to /var/log/fail2ban.log for Fail2ban v0.8.11

- Indicating no bans at all took place in the past three days.

Similar bugs are also reported by others in various places such as stack exchange, for example:

http://stackoverflow.com/a/20871940/1001513

http://serverfault.com/q/646193/89209

https://www.howtoforge.com/community/threads/fail2ban-fails-to-ban.37659/

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: fail2ban 0.8.11-1
ProcVersionSignature: Ubuntu 3.13.0-35.62-generic 3.13.11.6
Uname: Linux 3.13.0-35-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.6
Architecture: amd64
Date: Thu Jan 1 21:47:00 2015
PackageArchitecture: all
SourcePackage: fail2ban
UpgradeStatus: No upgrade log present (probably fresh install)

[Alistair Buxton (a-j-buxton)]

Possibly related:

https://github.com/fail2ban/fail2ban/issues/44

[Alistair Buxton (a-j-buxton)]

After a bit of testing it seems like the github issue actually is the problem. When auth.log gets rotated by logrotate, fail2ban no longer tracks changes to it, and so never takes any actions from that point on. To reproduce this:

1. Set up a server in a VM with sshd and install fail2ban. No config changes should be needed.
2. Run 'fail2ban-client set loglevel 4' to turn on debug logging (it should say "loglevel is DEBUG" or similar).
3. tail -f /var/log/fail2ban.log
4. tail -f /var/log/auth.log
5. ssh to server and enter wrong password. you should see activity in both log files when you do this.
6. Force log rotation: /usr/sbin/logrotate --force /etc/logrotate.conf - this should truncate auth.log
7. Try to ssh with wrong password again. You should still see the auth warning in auth.log but fail2ban will not notice any changes until you restart the fail2ban service.

[Alistair Buxton (a-j-buxton)]

Seems like that's not the problem after all...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[martin lantz (martin-lantz)]

I had this problem on Ubuntu 14.04 with Fail2Ban v0.8.11.

Using "backend = polling" in /etc/fail2ban/jail.local seems to work for me.

see discussion at; https://github.com/fail2ban/fail2ban/issues/878

[Seth Arnold (seth-arnold)]

Thanks for taking the time to report this bug and helping to make Ubuntu better. We appreciate the difficulties you are facing, but this appears to be a "regular" (non-security) bug. I have unmarked it as a security issue since this bug does not show evidence of allowing attackers to cross privilege boundaries nor directly cause loss of data/privacy. Please feel free to report any other bugs you may find.

[Alistair Buxton (a-j-buxton)]

This bug allows attackers to bypass security provisions designed to prevent brute-force attacks - which is the only purpose of the software.

[Bjoern Voigt (bjoern)]

I installed fail2ban 0.8.13-1 from Ubuntu Utopic on Ubuntu Trusty. With this version, Fail2Ban works as expected e.g. bans IP adresses with SSH jail.

[Ro (robert-markula)]

I'd second the downvote as an security issue.

Fail2ban's purpose is to block brute-force attacks. People installing this software rely on it to do just that. But immediately after the first logrotate, fail2ban will silently stop blocking brute-force attacks, letting attackers cross security boundaries that have been set up to do exactly that. All this while the user thinks itself safe.

Brute-force SSH attacks are extremely common and can lead to system break-ins sooner or later.

A security software that silently blacks out without any indication or warning of doing so, is dangerous. And a bug provoking this behaviour is a security issue after all.

I would not dare to count how many users out there are not aware that their fail2ban installation isn't working as expected, involuntarily facilitating brute-force attacks.

[Random User (dorh-kn)]

I have the exact same problem. I reproduced and observed the same behavior with logrotate as stated above. Polling solved the problem as stated.

But:

A few weeks ago my Ubuntu 14.04 server had strange problems and errors. I didn't have time to figure it out and made a clean install.
Yesterday I was shocked that my server had hundreds of attacks from same ip's and none of them are banned as usual. I do not usually check my logs as i rely on fail2ban. I am using fail2ban for years without any occasions. This is the most simple, primary and working solution on servers. Now i am sure that fail2ban stopped working after an update way before and some kind of attack caused my server to break.

This is a critical security issue. Comments above explain it clearly that this is a severe security issue. The purpose of fail2ban is security. I can repeat it over and over. This is a security bug. Importance is severe and critical not 'undecided' ! I can't believe that this is not taken seriously. 8 months since bug report and no body cares. This is unacceptable. This is the LTS version of Ubuntu server. I still can't believe it.

How can u say that this is a regular bug? If fail2ban does not work, an attacker can brute force a wordpress account, inject php code and cause loss of data and privacy. I don't even mention ssh. There are many more scenarios. Many people rely on fail2ban as it is a simple and effective security solution. I can't even imagine users unaware this situation.

Bugs can happen; but having a confirmed security bug report and not taking it seriously for 8 months is unacceptable. I spent so much time to figure the problem and i couldn't. I am not an expert. If this bug was not reported, i would not have figured out the problem and solve it.

How can i trust this distro anymore, who knows how many other critical issues that are not taken seriously !
It is time to move on to another distro...

I sincerely thank these 4 people who reported this bug, suggested solutions and have taken this issue seriously.