Search
  • Leonardo M. Falcon

Hunting on Sysmon events with Jupyter Notebooks - Environment setup

Updated: Aug 28

This post it's the first of a series where we will explore the capabilities of Sysmon and how we can leverage this tool together with Jupyter notebooks and big data Python libraries like Pandas to hunt for indicators of compromise or behaviors which are the result of the activity of advanced adversaries in a network. In this post we will provide some background information about Sysmon and the steps needed to install and configure a Sysmon hunt environment for research and development purposes.

About Sysmon

System Monitor (Sysmon) is a tool developed by Mark Russinovich at Microsoft. It is a Windows driver and service that logs system activity and saves it to the Windows event log. It provides detailed information about process creations, network connections, changes to files, among many other things. Currently, Sysmon provides up to 23 different types of events that are useful for monitoring various aspects of Windows system behavior. The Sysmon agent can be installed in persistent mode and will restart every time the system reboots. The Sysmon events are generated in Windows devices and can be collected in a central location to be used subsequently for continuous monitoring, Threat Hunting, and Incident Response.

Sysmon includes the following capabilities:

  • Logs process creation with full command line for both current and parent processes.

  • Records the hash of process image files using SHA1 (the default), MD5, SHA256 or IMPHASH.

  • Multiple hashes can be used at the same time.

  • Includes a process GUID in process create events to allow for correlation of events even when Windows reuses process IDs.

  • Include a session GUID in each events to allow correlation of events on same logon session.

  • Logs loading of drivers or DLLs with their signatures and hashes.

  • Logs opens for raw read access of disks and volumes

  • Optionally logs network connections, including each connection’s source process, IP addresses, port numbers, hostnames and port names.

  • Detects changes in file creation time to understand when a file was really created. Modification of file create timestamps is a technique commonly used by malware to cover its tracks.

  • Automatically reload configuration if changed in the registry.

  • Rule filtering to include or exclude certain events dynamically.

  • Generates events from early in the boot process to capture activity made by even sophisticated kernel-mode malware.

More information about Sysmon it's available in Microsoft's website.

Sysmon provides good theoretical coverage of the MITRE ATT&CK matrix, as shown in the diagram below.


Developed by @olafhartong (Follow him on Twitter)

It is possible to enhance further the coverage with additional attack correlation logic targeting specific Sysmon events. For example, using the Sysmon event ID 1 (Process creation), we can correlate different anomalies in the command line used by the system's running processes. Some examples are the execution of PowerShell commands with suspicious command line parameters or obfuscation and the execution of internal Windows commands with specific parameters to gather system information.


Installation of the hunting environment

To start exploring and hunting Sysmon data, you need to set up an environment to generate this data, store it, and process it. Our research and development environment uses Winlogbeat to forward the Sysmon logs, Logstash for filtering/parsing and an Azure data lake to store them. We have a virtual Jupyter Lab environment hosted in a powerful ESXi server for hunting purposes. A simpler setup can leverage Winlogbeat and Logstash on top of VMWare Workstation or VirtualBox to store and consume the log files locally in your own computer. Below we are proposing a relatively simple setup that can be reproduced in a commodity computer with enough RAM and CPU.

Requirements

  • We recommend using a laptop/workstation with a minimum of 16GB of RAM and a 64-bit x86 Intel or AMD Processor from 2011 or later.

  • VMWare Workstation or VirtualBox

  • A newly installed and updated Ubuntu 18.04 virtual machine

  • Enabled WSL and Ubuntu 18.04 environment installed

In our examples, we use VMWare Workstation as a virtualization engine, but theoretically, everything described in this article should work using VirtualBox as well.

Logstash server configuration

You can install Logstash in Ubuntu 18.04 following the Elastic guide. After the installation, follow the steps below to configure Logstash and prepare it for Jupyter.


1. Configure Logstash

  • Create a folder to store the Sysmon logs received by Logstash

$ sudo mkdir /var/sysmon-logs
$ sudo chmod 777 /var/sysmon-logs

  • Create a new Logstash configuration file for Winlogbeat

$ sudo vim /etc/logstash/conf.d/winlogbeat.conf

Insert the lines below:

input {
	beats {
 		port => 5044
 	}
}
 
output {
	if [agent][type] == "winlogbeat" {
 		file {
 			path => "/var/sysmon-logs/winlogbeat-%{+YYYY-MM-dd-HH}.json"
 		}
	}
}

With this configuration Logstash will create local log files that will be rotated on hourly basis.

  • Restart the Logstash service

$ sudo systemctl restart logstash

2. Create an anonymous share to access the Sysmon logs stored by Logstash

  • Install samba-server

$ sudo apt install tasksel
$ sudo tasksel install samba-server

  • Next, add the following lines into the Samba configuration file using your favorite text editor

$ sudo nano /etc/samba/smb.conf
[sysmon-logs]
 comment = Public anonymous share for Sysmon logs
 path = /var/sysmon-logs
 browsable =yes
 create mask = 0660
 directory mask = 0771
 writable = yes
 guest ok = yes

  • Restart the Samba service

$ sudo systemctl restart smbd

  • Test the access to the new share

$ smbclient -L localhost

The output should look similar to the one below.



3. Map the new share as a drive in your host system

Note: In this example we are using Windows 10 as host system.

  • Map the share in a new drive


Use the IP address allocated to your own Logstash virtual server. Use the credentials below to connect:

Username: anonymous

Password: anonymous

Install Sysmon and Winlogbeat

In this section we will cover the steps needed to install and configure Sysmon and Winlogbeat in the Windows system that will be the target for monitoring/hunting.

1. Install Sysmon

  • Download Sysmon from here and extract the contents of the zip file

  • Download the SwiftOnSecurity configuration file for Sysmon from here and save it in the same folder as Sysmon

  • Open a Windows command prompt as admin and execute the command below to install Sysmon

> Sysmon64.exe -accepteula -i sysmonconfig-export.xml

2. Install Winlogbeat

  • Download the Winlogbeat zip file from the downloads page

  • Extract the contents into C:\Program Files

  • Rename the winlogbeat-<version> directory to Winlogbeat

  • Open a PowerShell prompt as an Administrator (right-click on the PowerShell icon and select Run As Administrator).

  • From the PowerShell prompt, run the following commands to install the service.

PS C:\Users\Administrator> cd 'C:\Program Files\Winlogbeat'
PS C:\Program Files\Winlogbeat> .\install-service-winlogbeat.ps1

  • Configure Winlogbeat

Edit the file C:\Program Files\Winlogbeat\winlogbeat.yml and update it according to the pictures below.





  • Start the Winlogbeat service

PS C:\Program Files\Winlogbeat> Start-Service winlogbeat

Install Jupyter Lab

If you haven't done it, follow this guide to install WSL in your Windows 10 host system. After WSL it's installed, go to the Microsoft store and install Ubuntu 18.04.



Open a new Ubuntu WSL console and follow the steps below to install a new Jupyter Lab environment.

  • Create a new Python 3 virtual environment for Jupyter

$ cd ~
$ virtualenv -p python3.6 jupyter

  • Activate the new environment

$ source jupyter/bin/activate

  • Install all the required Python libraries

(jupyter)$ pip3 install jupyterlab pandas plotly scipy sklearn matplotlib

  • Generate Jupyter configuration file

(jupyter)$ jupyter notebook --generate-config

  • Edit the config file and set the password for Jupyter

(jupyter)$ vim /home/[USER]/.jupyter/jupyter_notebook_config.py
## Hashed password to use for web authentication.
#
# To generate, type in a python/IPython shell:
#
# from notebook.auth import passwd; passwd()
#
# The string should be of the form type:salt:hashed-password.
c.NotebookApp.password = u'HASHED PASSWORD'

  • Install Jupyter Lab extension for data visualization using Plotly

(jupyter)$ jupyter labextension install jupyterlab-plotly@4.8.1
(jupyter)$ jupyter labextension install plotlywidget@4.8.1

  • Start Jupyter

(jupyter)$ jupyter lab

  • Open a new browser window and navigate to the URL below to access Jupyter

http://localhost:8888/

  • Login to the web console using the password created previously


If the installation went well you should be able to access the Jupyter console.


Next steps

In our next post, we will start using our new hunt environment to explore and hunt on Sysmon data. We will focus on process execution events (Sysmon event ID 1), and we will provide some useful hints on how to use Python playbooks to catch advanced adversaries.


Stay tuned!

126 views
  • LinkedIn Social Icon

Prague, Czechia

©2020 Falcon Guard. All rights reserved.