apache-fakegooglebot jail bans real Google bots

[Mikkel Kirkgaard Nielsen]

On a mostly fresh Ubuntu 16.04 server with fail2ban the apache-fakegooglebot jail falsely detect accesses by authentic google bots as a fakes. This is because the ignore script at /etc/fail2ban/filter.d/ignorecommands/apache-fakegooglebot depends on fail2ban python code installed only for python3 but the shebang dictates it being run using the default python at /usr/bin/python which is 2.7.

$ /etc/fail2ban/filter.d/ignorecommands/apache-fakegooglebot 66.249.69.54
Traceback (most recent call last):
  File "/etc/fail2ban/filter.d/ignorecommands/apache-fakegooglebot.distrib", line 32, in <module>
    is_googlebot(process_args(sys.argv))
  File "/etc/fail2ban/filter.d/ignorecommands/apache-fakegooglebot.distrib", line 17, in process_args
    from fail2ban.server.filter import DNSUtils
ImportError: No module named fail2ban.server.filter

Forcing python3 gives the expected result (66.249.69.54 is a google webcrawler and should be ignored):
$ /usr/bin/python3 /etc/fail2ban/filter.d/ignorecommands/apache-fakegooglebot 66.249.69.54 && echo $?
0

Explicitly calling python3 in the shebang should solve it (distributed script backed up in .distrib):
$ diff /etc/fail2ban/filter.d/ignorecommands/apache-fakegooglebot{.distrib,}
1c1
< #!/usr/bin/python
---
> #!/usr/bin/python3

A more correct fix detecting the python version used by fail2ban was committed upstream almost a year ago (apache-fakegooglebot seems to be the only use case); https://github.com/fail2ban/fail2ban/issues/1506

Error in /var/log/fail2ban.log
2017-08-02 13:50:16,010 fail2ban.action [1565]: ERROR /etc/fail2ban/filter.d/ignorecommands/apache-fakegooglebot 66.249.69.54 -- stdout: b''
2017-08-02 13:50:16,010 fail2ban.action [1565]: ERROR /etc/fail2ban/filter.d/ignorecommands/apache-fakegooglebot 66.249.69.54 -- stderr: b'Traceback (most recent call last):\n File "/etc/fail2ban/filter.d/ignorecommands/apache-fakegooglebot", line 32, in <module>\n is_googlebot(process_args(sys.argv))\n File "/etc/fail2ban/filter.d/ignorecommands/apache-fakegooglebot", line 17, in process_args\n from fail2ban.server.filter import DNSUtils\nImportError: No module named fail2ban.server.filter\n'
2017-08-02 13:50:16,010 fail2ban.action [1565]: ERROR /etc/fail2ban/filter.d/ignorecommands/apache-fakegooglebot 66.249.69.54 -- returned 1
2017-08-02 13:50:16,010 fail2ban.filter [1565]: INFO [apache-fakegooglebot] Found 66.249.69.54
2017-08-02 13:50:16,770 fail2ban.actions [1565]: NOTICE [apache-fakegooglebot] Ban 66.249.69.54

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial

$ apt-cache policy fail2ban
fail2ban:
  Installed: 0.9.3-1
  Candidate: 0.9.3-1
  Version table:
 *** 0.9.3-1 500
        500 http://dk.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        500 http://dk.archive.ubuntu.com/ubuntu xenial/universe i386 Packages
        100 /var/lib/dpkg/status

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Anthony Geoghegan (anthony-geoghegan)]

Confirming this bug also affects users of Ubuntu 16.04.4 LTS - which has the same version of fail2ban (0.9.3-1).