• Home
  • About Us
  • Contact
  • Privacy Policy

LinuxAdmin.io

  • Home
  • Tutorials
    • System Administration
    • Linux Tutorials
    • Web Hosting
    • WordPress
    • Virtualization
    • Troubleshooting
    • OpenStack

Blocking Bad UserAgents with ModSecurity and Fail2ban

fail2ban

Many bots crawl websites that do nothing to generate positive traffic for the site. They just use up available resources and bandwidth of the server. You can reduce this drastically by implementing blocking using ModSecurity to detect the bad agents and then fail2ban to block them in iptables for a period of time.  This guide assumes you already have ModSecurity Installed. If you do not, follow our guide to get it installed then proceed with this.

Configure ModSecurity to Block  User Agents

In our Apache configuration setup, we already have a include directory for ModSecurity rules

Include /etc/httpd/conf/modsecurity.d/rules/*.conf

So we are going to create a new .conf to start detecting agents

nano /etc/httpd/conf/modsecurity.d/rules/block_user_agents.conf

We are going to add the following and save the file

SecRule REQUEST_HEADERS:User-Agent "@pmFromFile badbots.txt" "id:350001,rev:1,severity:2,log,msg:'BAD BOT - Detected and Blocked. '"

We are then going to create the list of User Agents to be detected and blocked:

nano /etc/httpd/conf/modsecurity.d/rules/badbots.txt

And insert the following user agents. If you want to let some of these in feel free to edit the list as you see fit.

AhrefsBot
Anonymizer
Attributor
Baidu
Bandit
BatchFTP
Bigfoot
Black.Hole
Bork-edition
DataCha0s
Deepnet Explorer
desktopsmiley
DigExt
feedfinder
gamingharbor
heritrix
ia_archiver
Indy Library
Jakarta
Java
juicyaccess
larbin
linkdex
Missigua
MRSPUTNIK
Nutch
panscient
plaNETWORK
Snapbot
Sogou
TinEye
TwengaBot
Twitturly
User-Agent
Viewzi
WebCapture
XX
Yandex
YebolBot
MJ12bot
masscan
baidu
Yandex
RSSingBot
Scanbot
betaBot
DotBot
SemrushBot
mj12bot
FeedFetcher
seoscanners.net
Moreover
ltx71
inboundlinks.win
sitebot

Configure Fail2Ban

First you will need to install Fail2ban

yum  -y install fail2ban

After that has finished installing, you will want to create a new jail file

nano /etc/fail2ban/jail.local

Creating a local jail will allow the main fail2ban configuration to be updated with new updates.

[apache-modsecblock-badbots]
enabled = true
filter = apache-useragent
logpath = /var/log/httpd/error_log
action = iptables-multiport[name=apache-badbots, port="http,https", protocol=tcp]
 postback[name=BADBOT, port="http,https", protocol=tcp]
maxretry = 2
bantime = 172800
ignoreip = 127.0.0.0/8 10.0.0.0/8 192.168.1.0/24

Update ignoreip with any local IPs or any others you want to allow in regardless of the UserAgent. This allows each IP to access twice with the a UserAgent indicated in the list, after that it will be banned.

You will then want to create the failregex pattern

nano /etc/fail2ban/filter.d/apache-useragent.conf

And add the following

# Fail2Ban configuration file
#

[Definition]


# Option: failregex
# Notes.: Regexp to catch known spambots and software alike. Please verify
# that it is your intent to block IPs which were driven by
# abovementioned bots.
# Values: TEXT
#

failregex = [[]client <HOST>[]] ModSecurity: Access denied with code 406 .* [[]msg "BAD BOT - Detected and Blocked. "[]] .*$

# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
#

Go ahead and restart fail2ban and apache

service httpd restart

service fail2ban restart

You should now be able to watch the apache error log /var/log/httpd/error.log to see if any bans are picked up.

 

 

 

May 31, 2017LinuxAdmin.io
0 0 vote
Article Rating
Nginx Caching for Wordpress using fastcgi_cachePiwik Analytics on Nginx
You Might Also Like
 
mod_auth_token Installation In Apache
 
How To Setup SSH Key Authentication
Subscribe
Notify of
guest
guest
2 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Chris
Chris
5 years ago

Hi, very interesting approach – but have not managed to get rid of ahrefs bot so far with this method. Followed your instructions, albeit slightly modified since on Debian. Badbots are detected and written into the error.log alright, but fail2ban fails to pick on them … maybe something wrong with the regex? It looks good to me, but since I am zero on regex I might be overlooking something … Here a sample of what is written into the error.log: [Tue Nov 28 16:34:54.520349 2017] [:error] [pid 30083] [client 51.255.65.33:49402] [client 51.255.65.33] ModSecurity: Warning. Matched phrase “AhrefsBot” at REQUEST_HEADERS:User-Agent. [file… Read more »

0
Reply
LinuxAdmin.io
Author
LinuxAdmin.io
5 years ago
Reply to  Chris

Hello Chris,
It might be the format, have tried doing a fail2ban-regex on the jail and log file to confirm its matching the hit through mod security?

0
Reply
5 years ago 2 Comments Apache, Linux System Administration, Security, Web Hosting centos, linux7,132
Recent Posts
  • Laravel Installation Guide For CentOS
  • Openstack Services Explanation And Overview
  • OSSEC Intrusion Detection Installation On Centos 7
  • Configure ProFTPd for SFTP on CentOS
  • How To Check And Repair MyISAM Tables In MySQL
Most Commented
Hot Clone Linux Server with Rsync
Hot Clone A CentOS Server With Rsync
14 Comments
ngx_cache_purge module
Install The ngx_cache_purge Module In Nginx
8 Comments
piwik-nginx
Piwik Analytics on Nginx
8 Comments
Tags
linuxcentosLinux Performance Tuningsysadminkvmnetworkingmemcachedanalyticssystemd
About

We love Linux and are dedicated to creating Linux administration tutorials for System Administrators since 2016.

Most Viewed
Default Gateway
How To Configure A Default Gateway on CentOS
62,983 views
Zend Opcache
Setup and Optimize Zend OpCache
46,714 views
Install ffmpeg
FFMpeg Install On CentOS 7
20,909 views
Archives
Email subscription

Sign up for our newsletter to receive the latest news and event postings.

2018 © LinuxAdmin.io
wpDiscuz