Using the Splunk SDK for Python to Programmatically Save Alerts

Introduction

The following post is intended to be the first in a series that will explore practical approaches, leveraging modern software development concepts, to operationalize cyber threat intelligence and detection engineering for security operations center (SOC) functions.

The posts will jump around in order, but the goal is to provide usable content that practitioners can immediately adopt and an overall guide that will be delivered at the end of the series.

The series will focus on specific tools, however, the concepts should still be applicable for similar purposed tools. Additionally, the series will use generalized terms for cybersecurity functions and will likely not align with your organization’s naming conventions. The focus of these posts are on the practical approaches and resulting value and less on the nuanced lexicon.

While the content may expand, the following outlines the concepts that will be explored in this series:

  • Programmatically save Splunk alerts for backup and recovery.
  • Detect attacks against Splunk that impair defenses or when there is an unintentional data source change.
  • CI/CD cyber threat intelligence and detection pipelines.
  • Test-driven detection engineering.
  • Automating cyber threat intelligence collection and existing detection coverage analysis.

The Need to Programmatically Save Splunk Alerts

If you‘re part of a SOC function that manages detection content, how would you answer the following questions?

  • Does your organization have a centralized repository of all Splunk alerts?
  • Do you have visibility to the change history of an alert?
  • Is a single analyst tasked with modifying detection content?
  • Is the repository backed up?
  • Could you quickly restore an alert if it was deleted?

If you answered ‘No’ to any of these questions, you have an opportunity to implement automated controls over the detection content.

Detection content should be managed using a continuous software development approach. Specifically, leveraging a collaborative software development and version control platform like GitHub can address the gaps noted in the questions above.

Prerequisites

The following are prerequisites needed to programmatically save Splunk alerts:

  • A Splunk environment with an understanding of how to create alerts and manage accounts.
Free Splunk

Create a GitHub Repository

The first step to save Splunk alerts is to create a GitHub repository to manage the detection content.

  1. Create a new empty repository in GitHub using a Python .gitignore template. You can name this repository based on your organization’s naming conventions, but for the purpose of the post, splunk-sdk was used.
  2. Clone the repository to the system that will run the Splunk Enterprise SDK for Python and has access to your Splunk instance.
  3. Create two directories called python and searches.
  4. The Splunk SDK for Python imports a module from a local file utils.py as part of the sample code and should be copied into the python folder.

Change directory into the python folder:

cd python

Use wget to download the utils.py file:

wget https://raw.githubusercontent.com/splunk/splunk-app-examples/master/python/utils.py

Your directory structure should look like the following view below.

splunk-sdk/
├── python
│ └── utils.py
└── searches

Install the Splunk Enterprise SDK for Python

Splunk provides a verbose overview of the Splunk Enterprise SDK for Python. The Splunk SDK for Python allows for programmatic interaction with a Splunk environment. In this use case, we’ll leverage the Splunk SDK for Python to save our alerts.

To install the Splunk SDK and establish an initial connection:

  1. Install splunk-sdk using pip:
pip install splunk-sdk

2. Create a .env file in the python folder:

Splunk SDK can use a .env file to connect and authenticate to your Splunk environment. The .gitignore file will prevent this file from getting pushed to the GitHub repository, however, anyone with access to the server and file can view the credentials. This specific post will not cover environment variable best practices, but later posts will explore Splunk permissions and abuse mitigations. As Splunk notes:

Note: Storing login credentials in the .env file is only for convenience during development. This file isn’t part of the Splunk platform and shouldn’t be used for storing user credentials for production. And, if you’re at all concerned about the security of your credentials, enter them at the command line rather than saving them in this file.

Change directory into the python folder:

cd python

Change directory into the python folder:

nano .env

Paste the .env contents below and ensure you update the host, port, username, and password to reflect your environment:

# Splunk Enterprise host (default: localhost)
host=localhost
# Splunk Enterprise admin port (default: 8089)
port=8089
# Splunk Enterprise username
username=admin
# Splunk Enterprise password
password=changed!
# Access scheme (default: https)
scheme=https
# Your version of Splunk Enterprise
version=8.0
# Bearer token for authentication
#bearerToken=<Bearer-token>
# Session key for authentication
#sessionKey=<Session-Key>

3. Install additional dependencies for python-dotenv support:

pip install python-dotenv

The directory structure of your repository should now look like the following:

splunk-sdk/
├── python
│ ├── .env
│ └── utils.py
└── searches

Testing the Splunk SDK Connection

Prior to implementing code to save Splunk alerts, let’s test connecting to your Splunk instance using the Splunk SDK.

  1. Change directory into the python folder:
cd python

2. Run python:

python

3. From the python prompt, import the following modules:

import sys

If this fails, the Splunk SDK to not install correctly:

from splunklib.client import connect

If this fails, the utils.py file may not have saved correctly or is not in the python directory:

from utils import parse

4. From the python prompt, issue the following”

If this fails, the .env file may not be in the python directory:

opts = parse(sys.argv[1:], {}, ".env")

If this fails, the host port username or password variables in the .env file may be incorrect. Also, verify you the Splunk environment doesn’t have an access control list restricting access. A good approach is to log into the web UI using the account credentials used in the .env file.

 service = connect(**opts.kwargs)

If there are no errors returned, you likely have established a successful connection to your Splunk instance and can implement the code to programmatically save Splunk alerts.

Create a PATH Environment Variable

Prior to implementing code to save Splunk alerts, create a directory PATH environment variable that will be used in the code.

  1. Setup a directory path environment variable. This should be the full path used for the searches directory:
export SAVE_SEARCHES_PATH='full_directory_path'

Verify the path variable was setup correctly and results are returned:

printenv | grep SAVE_SEARCHES_PATH

Use the Splunk SDK to Save Alerts

With all of the prep-work completed to setup a GitHub repository, install the splunk-sdk and its dependencies, and verify connectivity to the Splunk environment, you can now run the code snippet below to save your Splunk alerts.

The Splunk SDK can be used to save a multitude of information and configuration settings for the alert, including the description, scheduling, and alert actions, however, for this post, we’ll focus on saving the alert name and search syntax only.

  1. Change directory into the python folder:
cd python

2. Create a file save_searches.py to run the Splunk SDK code:

nano save_searches.py

3. Copy and paste the following content into save_searches.py :

import os
import sys
from splunklib.client import connect
from utils import parse
def main():
opts = parse(sys.argv[1:], {}, ".env")
service = connect(**opts.kwargs)

savedsearches = service.saved_searches

search_dir = (os.environ['SAVE_SEARCHES_PATH'])
os.chdir(search_dir)

for savedsearch in savedsearches:
f = open('%s' % savedsearch.name, 'wb')
f = open(str(savedsearch.name), 'a')
f.write(savedsearch["search"])
f.close()
if __name__ == "__main__":
main()

The following code will connect to Splunk and leverage the Splunk SDK splunklib.client.SavedSearches class for the collection of saved searches.

After changing to a dedicated output directory, the search content for each saved alert in the collection is saved to a matching filename.

4. Execute the code:

python save_searches.py

If everything executed correctly, you should now see your alerts, with the search syntax, in the searches directory:

splunk-sdk/
├── python
│ ├── .env
│ ├── save_searches.py
│ └── utils.py
└── searches
└── Test Alert

5. With the alerts saved, the changes should be committed and pushed to GitHub. Additionally, the save_searches.py can be setup on a cron job to automatically pull down new alerts and modifications to existing alerts.

Conclusion

The post provides a basic introduction to the Splunk SDK for Python and how to programmatically save an alert and search syntax. Additionally, leveraging GitHub to centralize detection content will provide the initial foundation to build CI/CD detection pipelines. For example and in subsequent posts, we’ll explore how to use the Splunk SDK to perform detection rule integrity checking, provide results to evaluate fidelity, and push detection content changes to Splunk.

Thank you for taking the time to read through this!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store