Web Shells. Additional Detection Strategies with Splunk Hands-On Labs.
An overview of a web shell with a hands-on lab was covered in the post Web Shells. An Introduction and Detection Strategies with YARA Hands-On Labs. The following post will expand on this initial lab to provide additional detection strategies using Splunk as a security information and event management (SIEM). The post Enterprise Visibility with Splunk. The Building Blocks for Incident Detection. provides a lab on deploying Splunk Enterprise.
Setup the Splunk Universal Forwarder
The Splunk Universal Forwarder (UF) will need to be install on the webshell-webserver instance to collect relevant logs. The following logs will be utilized to build detection signatures for the weevly web shell:
- Apache web server logs for the WordPress web site
- Ubuntu operating system command output
- The Linux Audit daemon (auditd) logs
Download the Splunk UF
Log into your Splunk Enterprise environment. From the top menu bar, select Universal Forwarder.
From the Universal Forwarder page, click the first link to download the Splunk UF.
Select the Linux tab and download the .deb package. When prompted to download, cancel the local download and click the option to Download via Command Line (wget). At the time of this post, the Linux Splunk UF was on version 8.0.6.
On macOS open the terminal app and ssh as root to the webshell-webserver IP address. On Windows, utilities like putty can be leveraged to connect to the IP address.
ssh root@$IP_ADDRESSFrom the webshell-webserver instance, download the Splunk UF.
root@webshell-webserver:
wget -O splunkforwarder-8.0.6-152fb4b2bb96-linux-2.6-amd64.deb 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.6&product=universalforwarder&filename=splunkforwarder-8.0.6-...-linux-2.6-amd64.deb&wget=true'Install the Splunk UF
Install the Splunk using the dpkg command.
root@webshell-webserver:
dpkg -i splunkforwarder-8.0.6-...-linux-2.6-amd64.debNavigate to the /opt/splunkforwarder/bin directory and start Splunk. Read and accept the license.
root@webshell-webserver:
cd /opt/splunkforwarder/bin/
./splunk start
Do you agree with this license? [y/n]: yEnter a Splunk UF administrator username and password.
Please enter an administrator username: splunk
Please enter a new password:Once the credentials are entered, the Splunk UF will finish installing.
The Splunk UF does not auto start on boot. Issue the following command to have the Splunk UF start on boot.
root@webshell-webserver:
/opt/splunkforwarder/bin/splunk enable boot-startSetup Splunk UF Credential Package
A credential package will need to be configured with the Splunk UF. Download the Universal Forwarder Credentials.
Copy the credential package to the webshell-webserver instance. The credential package may have a different file name, but will end with a .spl file extension. Run the splunk install app command to install the credential package.
root@webshell-webserver:
cd /root
/opt/splunkforwarder/bin/splunk install app splunkclouduf.spl
Apache Web Server Logs
To send Apache web server logs to Splunk Enterprise, an add-on app and a Splunk UF Technology Add-on (TA) can installed and configured. From the top menu bar, clicks Apps and select Find More Apps.
From the App search, find “Splunk Add-on for Apache Web Server” and click Install. You will need to enter your Splunk account credentials.
The Splunk UF TA will still need to be installed and configured on the webshell-webserver instance, but some additional changes will be made for the Splunk Enterprise add-on.
A unique index will be created for Apache logs. Having a unique index for a log source can support consistent alert creation, operational management of an index, and cost analysis of the data source. From the top menu click Settings and select Indexes under the Data section.
Click New Index to create a new Splunk index. Enter apache as the index name, 0 for the max raw data size, and 180 for searchable time (days). Save the new index.
From the top menu bar, click Apps and select Manage Apps.
Search for the Apache app and click View objects.
From the All configurations section, click access_log_event.
In the Search string input, add the following statement to include the newly created Apache index in the add-on and save the configuration.
OR index=apache
The Splunk UF TA needs to be downloaded and configured on the webshell-webserver instance. Download the app and copy it to the webshell-webserver instance.
Move the downloaded Splunk add-on to the /opt/splunforwarder/etc/apps directory. Run the tar command to extract the files.
root@webshell-webserver:
mv splunk-add-on-for-apache-web-server_200.tgz /opt/splunkforwarder/etc/apps/cd /opt/splunkforwarder/etc/apps/
tar -xvf splunk-add-on-for-apache-web-server_200.tgz
Navigate to the newly created directory Splunk_TA_apache, make a new directory call local, and navigate to this directory.
root@webshell-webserver:/opt/splunkforwarder/etc/apps
cd Splunk_TA_apache
mkdir local
cd localFor a full overview, Splunk docs provides an overview on the Splunk Add-on for Apache Web Server.
In the local directory, run the nano command and create a new file called inputs.conf.
root@webshell-webserver:
nano inputs.confAdd the following to the inputs.conf file to configure the Apache access and error logs. This configuration will send these log source types to the Apache index created above.
[monitor:///var/log/apache2/error.log]
index=apache
sourcetype=apache:error
disabled = 0[monitor:///var/log/apache2/access.log]
index=apache
sourcetype=access_combined
disabled = 0
Restart the Splunk UF.
root@webshell-webserver:
/opt/splunkforwarder/bin/./splunk restartFrom a local browser, access the WordPress administrative panel to generate a local log event. From the Splunk Enterprise Search & Reporting app, enter the query below to verify that logging is working correctly.
index=apache sourcetype=access_combined
Ubuntu Operating System Logs
To send Ubuntu operating system logs to Splunk Enterprise, the Splunk Add-on for Unix and Linux add-on app can installed and configured. Repeat the steps above to search and install the add-on. For this lab a new index will not be created, however, in a production deployment it would be advised to create a unique index.
The Splunk Add-on for Unix and Linux TA will need to be downloaded and configured on the webshell-webserver instance. The documentation about this add-on can be found here.
From the webshell-webserver instance, move the Splunk Add-on for Unix and Linux TA file to the /opt/splunkforwarder/etc/apps directory and extract the files using the tar command.
root@webshell-webserver:
mv splunk-add-on-for-unix-and-linux_820.tgz /opt/splunkforwarder/etc/apps/cd /opt/splunkforwarder/etc/apps/
tar -xvf splunk-add-on-for-unix-and-linux_820.tgz
Navigate to the newly created Splunk_TA_nix directory and make a new local directory. Copy the inputs.conf file in the default directory to the local directory. Navigate to the local directory.
root@webshell-webserver:
cd Splunk_TA_nix/
mkdir local
cp default/inputs.conf local/inputs.conf
cd local/From the local directory edit the inputs.conf file using the nano command.
root@webshell-webserver:
nano inputs.confSet disabled = 0 for all items under the Metric Input and Event Inputs stanzas. This will send the logs to the associated configuration items to Splunk Enterprise.
Restart the Splunk UF.
root@webshell-webserver:
/opt/splunkforwarder/bin/./splunk restartSetup the Linux Audit System
The Linux Audit System can be configured to log events based on a rules file. To enable this capability, auditd, which is the Linux Audit daemon, can be installed via apt by issuing the following command.
root@webshell-webserver:
sudo apt install -y auditdApache runs under the www-data user and the user id will be required to uniquely log events within this user context. Run the id command with the switch “-u” and www-data to identify the id value.
root@webshell-webserver:
id -u www-data33
To add auditd rules, navigate to the /etc/audit/rules.d directory. Run the nano command to edit the audit.rules file.
root@webshell-webserver:
cd /etc/audit/rules.d
nano audit.rulesAdd the following items to the audit.rules file. The first rule will log all write events in the /var/www/html directory, where WordPress is installed, with a unique key called “auditd_web”. The remaining rules will log all commands issued by the www-data user with a unique key of “auditd_command”.
-w /var/www/html -p w -k auditd_web
-a always,exit -F arch=b64 -S execve -F uid=33 -F -k auditd_command
-a always,exit -F arch=b32 -S execve -F uid=33 -F -k auditd_commandRestart the auditd service.
root@webshell-webserver:
service auditd restartTo verify the rules are in place, run the auditctl command.
root@webshell-webserver:
auditctl -l
Linux Audit System Logs
To send Linux Audit System logs to Splunk Enterprise, the Linux Auditd Technology Add-On app can installed and configured. Repeat the steps above to search and install the add-on. For this lab a new index will not be created, however, in a production deployment it would be advised to create a unique index.
The Linux Auditd Technology Add-On TA will need to be downloaded and configured on the webshell-webserver instance. The documentation about this add-on can be found here.
From the webshell-webserver instance, move the Linux Auditd Technology Add-On TA file to the /opt/splunkforwarder/etc/apps directory and extract the files using the tar command.
root@webshell-webserver:
mv linux-auditd-technology-add-on_310.tgz /opt/splunkforwarder/etc/apps/cd /opt/splunkforwarder/etc/apps/
tar -xvf linux-auditd-technology-add-on_310.tgz
For this TA, restart the Splunk UF.
root@webshell-webserver:
/opt/splunkforwarder/bin/./splunk restartTo verify the audit.log source is correctly logging, enter the following query in the Splunk Enterprise Search & Reporting app and ensure there are events.
source="/var/log/audit/audit.log"
Web Shell Lab Continued
As a continuation of the web shell lab, repeat the web shell upload from the Web Server Exploitation step in the lab Web Shells. An Introduction and Detection Strategies with YARA Hands-On Labs.
With the weevely web shell successfully uploaded to the webshell-webserver instance, SSH into the webshell-rsync instance and navigate to the weevely3 directory. From the weevely3 directory, run the following command to connect to the web shell.
root@webshell-rsync:weevely3
./weevely.py http://webshell-webserver_IP_ADDRESS/wp-content/plugins/webshell.php labpassword
With the weevely web shell uploaded and a session established, a Splunk query can be run to determine if the event was logged.
Initial Web Shell Upload Event
While it may be difficult to build a detection signature from a file upload log, the information is critical during the incident response process to identify all malicious activity.
For the lab, the web shell name is known and can be searched using the following query against the apache index. With the log, an actor’s source IP address (clientip field) and time stamp can be identified.
index=apache sourcetype=access_combined webshell.php
Additionally, the following query can be run against the audit.log source. The audit log contains a CREATE event under the nametype field.
source="/var/log/audit/audit.log" webshell.php
Weevely Web Shell Use
The weevely web shell has multiple modules that can be used for post compromise techniques.
If the target PHP settings don’t allow to execute external commands, Weevely uses
PHP>prompt and emulates commands such ascd,cat,curl,nanoin order to provide a functional shell access to the server.
From the weevely session, type :help to view all available modules.
For the lab, run the command :audit_phpconf to audit the PHP configuration file.
From the Apache logs, a POST request to the web shell is the only event. This is expected and from a web shell command perspective, will provide limited detection value.
An organization may be able to implement a detection signature that evaluates a unique IP address, conducting a certain count of POST requests to a .PHP file, over a defined period of time as an indicator of an actor’s interaction with a web shell. This will likely require up front tuning to exclude legitimate processes and identify an appropriate count threshold. The following Splunk query can be used as a starting point.
index=apache earliest=-1h method=POST uri_path="*.php" status="200"
| stats count by clientip
| search count > 50Given weevely is interacting with the operating system, evaluating the audit logs will likely give us additional detection signature creation opportunities. When auditd was configured, the www-data user (id 33) was configured and given a unique key of “auditd_command”. Using these values in the Splunk query, all log events at the time of weevely command can be viewed.
source="/var/log/audit/audit.log" uid=33 key="auditd_command"The event below should be logged. The event shows dash and sh interaction.
While this would need to be vetted against production web sites, there were no similar events during the normal operation of the WordPress web site and all similar events could be attributed to weevely modules. Using the extracted fields, a Splunk query can be created to identify the specific www-data user (uid=33), the unique key (key=”auditd_command”) and both the sh (comm=”sh”) and dash (command=”/bin/dash”) commands being issued. The Splunk query can be setup to run on a regular schedule to identify weevely web shell activity.
source="/var/log/audit/audit.log" uid=33 key="auditd_command "comm="sh" command="/bin/dash"MITRE ATT&CK
The weevely web shell also allows operating system commands to be input. Using the methods above, the technique System Owner/User Discovery can be evaluated using the weevely web shell to build detection signatures.
Adversaries may attempt to identify the primary user, currently logged in user, set of users that commonly uses a system, or whether a user is actively using the system. They may do this, for example, by retrieving account usernames or by using OS Credential Dumping. The information may be collected in a number of different ways using other Discovery techniques, because user and username details are prevalent throughout a system and include running process ownership, file/directory ownership, session information, and system logs. Adversaries may use the information from System Owner/User Discovery during automated discovery to shape follow-on behaviors, including whether or not the adversary fully infects the target and/or attempts specific actions.
Utilities and commands that acquire this information include
whoami. In Mac and Linux, the currently logged in user can be identified withwandwho.
It is unlikely that the www-data user would directly issue the commands whoami, w and who and the behavior would be suspicious. This behavior would be suspicious beyond just the weevely web shell, but this software can be used to generate log events and build detection signatures.
From the weevely web shell prompt, issue the whoami, w and who commands. Run a Splunk query to look for these commands.
source="/var/log/audit/audit.log" uid=33 key=auditd_command whoami OR w OR whoThe Splunk query will return results for each command issued through weevely under the www-data user.
A Splunk query can now be refined to match the specific procedures for the System Owner/User Discovery technique.
source="/var/log/audit/audit.log" uid=33 key=auditd_command
(comm="whoami" AND exe="/usr/bin/whoami") OR
(comm="w" AND exe="/usr/bin/w.procps") OR
(comm="who" AND exe="/usr/bin/who")Closing Remarks.
Cybersecurity analysts should be familiar with Apache, Linux OS, and auditd log aggregation and SIEM capabilities as a detection strategy for web shells. In a subsequent post for web shell detection, operationalizing logging YARA detection matches to alert in a SIEM will be explored.
If you made it this far, thanks for reading! Any feedback is always appreciated.
