Dynamically Block IPs Enumerating Your Web Site

Author: Chris Huey

Date: May 30, 2022

This article from chueyise.com provides an innovative approach for monitoring your Apache web access logs for suspicious activity and dynamically blocking nefarious source IPs using the iptables host-based firewall. The implementation involves the following:

Configuring Apache to allow easy identification of the HTTP Status Code within log files
Installing and configuring Syslog-NG to identify suspicious web activity within Apache log files
Installing and enabling the iptables host-based firewall and ipset framework
Deploying a simple, custom service (using systemd) to recognize suspicious web activity identified by Syslog-NG and dynamically configure iptables to block the source IP address for five (5) minutes

Prerequisites:

Apache 2.4 web server installed and fully functional
Linux distro installed that is forked to Red Hat Enterprise Linux (RHEL). Examples include: CentOS, AWS Linux, Fedora, Rocky Linux

Here’s how this implementation works as prescribed in Steps 1 through 5.

Step 1: Configure Apache LogFormat

Modify the Apache LogFormat setting to allow the HTTP Status Code to be easily recognized by a log monitoring tool (see Step 2). The HTTP Status Code is reflected in the LogFormat stanza as ‘%>s’. Below are steps to identify and modify the HTTP Status Code format:

As root from the command line, traverse to the Apache configuration file location. Normally the location is /etc/httpd:
cd /etc/httpd
Identify the Apache config files containing the LogFormat stanza.
grep -r –include=”*.conf” -i logformat
For each configuration file containing LogFormat, ensure there is a %>s field and add ‘::’ preceding the field. For example:
LogFormat “%a [%r] ::%>s \”%{Referer}i\” \”%{User-Agent}i\” %b” common
Restart the Apache
service systemctl restart httpd

Step 2: Install and Configure Syslog-NG

The objective of this step is to monitor the web server access logs, identify HTTP Status Codes of 4XX, and copy the logs to a dedicated log file of 4XX log messages. Using syslog-ng optional, but this author believes it is far superior to rsyslog. The rsyslog service is likely installed and implemented by default on your Linux system. If you desire to use syslog-ng instead of rsyslog, perform the following to disable rsyslog, and install and enable syslog-ng. If you choose to use rsyslog, read the information below and translate the configurations into an equivalent rsyslog implementation.

systemctl stop rsyslog
systemctl disable rsyslog

yum install -y syslog-ng
systemctl enable syslog-ng
systemctl start syslog-ng

Configure syslog-ng to monitor the Apache access logs for HTTP Status Codes of 400-499. These codes represent invalid (suspicious) activities against the web server site. Add the following stanzas to file /etc/syslog-ng/syslog-ng.conf. NOTE: ensure you provide a source stanza for each access log used by your apache configuration.

source s_accesslog { file(“/var/log/httpd/access_log”); };
source s_accessssllog { file(“/var/log/httpd/ssl_access_log”); };
destination d_400 { file(“/var/log/400.log”); };
filter f_400 { match (“^.+ \:\:4”); };
log { source(s_accesslog); filter(f_400); destination(d_400); };
log { source(s_accessssllog); filter(f_400); destination(d_400); };

Ensure the 400.log file exists and restart the syslog-ng service to enable the new log collection and processing rules.

touch /var/log/400.log
systemctl restart syslog-ng

Step 3: Install and Configure Iptables

Some Linux distributions enable firewalld by default. In this case, firewalld needs to be disabled as follows:

systemctl stop firewalld
systemctl disable firewalld

Install and enable iptables:

yum install iptables-services -y
systemctl enable iptables
systemctl start iptables

If you are not familiar with developing custom firewall rules, use this resource as a tutorial. Once iptables is baselined and fully functional, install the following rules within the rules file (/etc/sysconfig/iptables). The BLACKLIST_REJECT chain is used by the ipset framework (described below).

:BLACKLIST_REJECT – [0:0]
-A BLACKLIST_REJECT -m hashlimit –hashlimit-name BLACKLIST_HASH –hashlimit-mode srcip –hashlimit-upto 1/minute –hashlimit-htable-expire 60000 -j LOG –log-prefix “iptables BLACKLIST_REJECT ” –log-level 1
-A BLACKLIST_REJECT -j REJECT –reject-with icmp-host-prohibited

Restart the iptables firewall with the new rules:

iptables-restore < /etc/sysconfig/iptables

Install and initialize the ipset framework. The ipset framework allows dynamic filter rules to be defined within iptables.

yum install ipset -y
Create a tailored ipset configuration file containing:
create –exist blacklist hash:ip family inet hashsize 512 maxelem 65536 timeout 300
Save the file as /etc/ipset.conf. The file is used to create a BLACKLIST hash list used in Step 4. The BLACKLIST hash list is used to dynamically populate IP address. The addresses drop off the hash list after five (5) minutes.

Step 4: Deploy Custom Services to Block Suspicious IP Addresses

Use the following hyperlinked example files and customize as needed. This step enables all the services needed to continuously monitor for HTTP Status Codes 400-499 and dynamically block IP addresses performing suspicious activity.

400blocker: save as /usr/bin/400blocker.sh and execute chmod 750 /usr/bin/400blocker.sh
blacklist: save as /usr/bin/blacklist.sh and execute chmod 750 /usr/bin/blacklist.sh
400blocker.service: save as /etc/systemd/system/400blocker.service and execute the following:
systemctl daemon-reload
systemctl enable 400blocker
systemctl start 400blocker
blacklist.service: save as /etc/systemd/system/blacklist.service and execute the following:
systemctl daemon-reload
systemctl enable blacklist
systemctl start blacklist

If there were no errors from this step, your web server is actively monitoring the Apache access logs and blocking ip addresses performing web enumeration activities.

Step 5: Monitor For False-Positive Blocks and Fix

Monitor your apache logs to identify accesses resulting in 400 HTTP Status Codes that should not be blocked. As an example, if someone uses an iPhone to access your web site, the Apple browser will automatically attempt to access non-existent png files (apple-touch-icon.png, apple-touch-icon-precomposed.png) which will result in a 404 code. This is a legitimate access attempt that should not result in blocking the source IP address from further accesses to your site. This issue can be resolved by creating a nocares filter in your syslog-ng configuration file to ignore these types of accesses. Below are example settings to apply within the /etc/syslog-ng/syslog-ng.conf file.

filter f_nocares { not match (“systemd”) and
not match (“^.+\.png”) and
not match (“^.+Googlebot”) and
not match (“^.+bot/”) and
not match (“^.+\:\:408”);};

log { source(s_accesslog); filter(f_400); filter(f_nocares); destination(d_400); };
log { source(s_accessssllog); filter(f_400); filter(f_nocares); destination(d_400); };

Whenever you make changes to your /etc/syslog-ng/syslog-ng.config file, execute systemctl restart syslog-ng