Automatically Detect and Purge Hackers Gaining Web Shells

Author: Chris Huey
Date: June 4, 2022

What's a Web Shell

Hackers gain unauthorized access to your server by exploiting weaknesses normally existing from insecure configurations or unpatched components. They typically identify weaknesses by enumerating the system using automated scanning tools (bots, hackers, advanced persistent threats).

For example, you might notice numerous attempts to access pages on your web server that don’t exist. A scanning tool is attempting to find known attack vectors within components of the web server. At a minimum, a Web Application Firewall (WAF) like ModSecurity should be employed to add a layer of defense against pesky hackers. Additionally, you could make enumeration sessions more difficult using guidance in Dynamically Block Site Enumeration Sources Using Iptables.

When a hacker successfully exploits vulnerabilities in web-based applications, a shell is used to perform unauthorized actions on the exploited system. These actions execute as the service account owner of the exploited application. For example, the apache service account (with minimum access rights) owns the running apache httpd service. If a hacker exploits the httpd or php services, the commands performed within the shell run under the apache service account.

Once access is gained, the hacker typically attempts to escalate privileges (i.e., attempts to gain access to the root account). This Linux Privilege Escalation page provides an excellent overview of the privilege escalation process.

Solution to Detect and Purge Web Hackers

This article from chueyise.com provides an innovative solution to automatically and reliably recognize a hacker’s web shell, immediately purge the hacker, and notify the web administrator via text/email. The solution applies to servers running Linux with apache as the web server software. Below summarizes how the solution is implemented.

Configure the auditd service to generate ‘alert’ messages for typical commands used during privilege escalation activities
Monitor the auditd logs for ‘alert’ messages, and use information within the ‘alert’ message to terminate the chain of processes (but not the master process) associated with hacker activity
As a defense-in-depth feature, monitor processes owned by web-related service accounts (apache, mysql) for unknown processes that don’t trigger an auditd ‘alert’ log; immediately terminate any unknown processes and add suspicious IPs for existing network connections to the firewall block list (refer to Dynamically Block IPs Enumerating Your Web Site for details)
Generate an ‘attack’ log file containing information to assist in understanding how the successful attack was accomplished and help determine the required mitigations to protect against future exploits
Send an alert text message to the web administrator using Slack, visit How to Send a Message to Slack From a Bash Script

Optionally, you can create a permanent block list and augment the solution summarized above to automatically add IPs with active network connections (not part of your ecosystem) to the block list when hacker activity is identified. The IPs would remain in the list until you have a chance to assess and contain the breach. I implemented this feature on the chueyise.com web server, however it is not full-proof since the adversary can simply change IPs and attack again. Use guidance in Dynamically Block IPs Enumerating Your Web Site to understand the concept behind this implementation.

Indicators of a Successful Hack

Figure 1 shows hacker activity performed by the apache user (the process owner of the httpd and php-fpm services). This example process dump shows a hacker used a web shell to upload a privilege escalation script, set the script permission to allow execution, and then attempt to find the netcat utility. Figure 1 also highlights the process chain to show the sequence of processes associated with the hacker’s activity.

Step 1: Configure Auditd to Detect Hacker Shell Activity

This step involves configuring the linux auditd service to generate log messages for typical hacker actions. Use a best practice guideline to configure Linux systems auditing with auditd. Once auditd is enabled, configure the audit.rules file to record typical hacker shell activity but not for actions performed routinely by the service account. The audit.rules file is typically located in /etc/audit or /etc/audit/rules.d folder for Linux distributions.

Below is an example of a small subset of auditd rules related to potential hacker shell actions. In this example, the web server utilizes an apache web server (running under the apache service account) and a back-end mysql database (running under the mysql service account). The descriptive ALERT key (designated with the -k tag) establishes the search criteria for quickly identifying hacker activity.

IMPORTANT:
Dry run your rules to ensure ALERT logs are not generated for normal system activity. For example, mysql routinely performs ‘ls’ commands so a rule to log mysql ‘ls’ usage must be avoided.

The “Example subset of auditd rules” will result in the following auditd log when the find command in Figure 1 is executed.

Step 2: Implement the Hacker Detection Service

Step 2 involves implementing features to monitor the logs for ALERT keys and purge the hacker from the system using information within the ALERT logs. There are two basic components of the Hacker Detection Service: (1) a custom bash script which monitors the auditd logs for hacker activity (The Hacker Detection Script); and (2) a systemd configuration that enables this bash script as a service (The Hacker Detection Service). Each component is addressed below. Once the service is successfully deployed, it is important to test the service by simulating a successful hacker attack.

The Hacker Detection Script

Here’s an example of the Hacker Detection Script which includes comments to describe how it works.

The Hacker Detection Service

This component of the solution involves creating a unit configuration file with a name ending in ‘.service’ to define the Hacker Detection process for control and supervision by systemd. Below is an example of the Hacker Detection Service configuration file. The configuration file includes a stanza to automatically restart the service if it ever unexpectedly fails (Restart=on-failure).

Perform the following to enable the Hacker Detection Service for startup at system bootup.

cp [Hacker Detection Script file] /usr/bin/hackerdetect.sh
chmod 750 /usr/bin/hackerdetect.sh; chown root:root /usr/bin/hackerdetect.sh
cp [Hacker Detection Service configuration file] /etc/systemd/system/hackerdetect.service
systemctl daemon-reload
systemctl enable hackerdetect
systemctl start hackerdetect
systemctl status hackerdetect and verify you see ‘Active: active (running)’ in the output

Test the Hacker Detection Service

Create a simple php test file that simulates a hacker’s web shell. Below is an example of a php test file that triggers the Hacker Detection Service (IMPORTANT: Ensure your audit.rules file from Step 1 above includes an ALERT rule for ‘ps’ and ‘netstat’ AND the last php bash command is sleep 30 to simulate an open/active hacker web shell).

Test #1: Verify the auditd killchain function:

cp [Hacker Detection Service test file].php /var/www/html/hacked.php
chown apache:apahce /var/www/html/hacked.php
chmod 755 /var/www/html/hacked.php
Open a web browser and access https://[your web site]/hacked.php
Verify a text message/email was sent to the web administrator
cat /tmp/hacked.log verify sufficient data was logged for analysis of attack
cat /tmp/ignores.txt verify the PPIDs from the auditd ALERT messages exist

Test #2: Verify the unknown process termination function:

Open file /etc/passwd and temporarily change the apache service account configuration as follows:
apache:x:48:48:Apache:/usr/share/httpd:/bin/bash
Open a tty/ssh session and su to root: sudo su
Change user to the apache service account: su apache
Verify the apache user session is immediately killed (terminated)
Set the apache entry in the /etc/passwd file back to the original logon shell value (/sbin/nologin)