procmon-parser 0.3.13

Description:

procmonparser 0.3.13

Parsing Procmon files with Python




Procmon (https://docs.microsoft.com/en-us/sysinternals/downloads/procmon) is a very powerful monitoring tool for Windows,
capable of capturing file system, registry, process/thread and network activity.
Procmon uses internal file formats for configuration (PMC) and logs (PML).
Prior to procmon-parser, PMC files could only be parsed and generated by the Procmon GUI, and PML files
could be read only using the Procmon GUI, or by converting them to CSV or XML using Procmon command line.
The goals of procmon-parser are:

Parsing & Building PMC files - making it possible to dynamically add/remove filter rules, which can significantly
reduce the size of the log file over time as Procmon captures millions of events.
Parsing PML files - making it possible to directly load the raw PML file into convenient python objects
instead of having to convert the file to CSV/XML formats prior to loading.

PMC (Process Monitor Configuration) Parser
Usage
Loading configuration of a pre-exported Procmon configuration:
>>> from procmon_parser import load_configuration, dump_configuration, Rule
>>> with open("ProcmonConfiguration.pmc", "rb") as f:
... config = load_configuration(f)
>>> config["DestructiveFilter"]
0
>>> config["FilterRules"]
[Rule(Column.PROCESS_NAME, RuleRelation.IS, "System", RuleAction.EXCLUDE), Rule(Column.PROCESS_NAME, RuleRelation.IS, "Procmon64.exe", RuleAction.EXCLUDE), Rule(Column.PROCESS_NAME, RuleRelation.IS, "Procmon.exe", RuleAction.EXCLUDE), Rule(Column.PROCESS_NAME, RuleRelation.IS, "Procexp64.exe", RuleAction.EXCLUDE), Rule(Column.PROCESS_NAME, RuleRelation.IS, "Procexp.exe", RuleAction.EXCLUDE), Rule(Column.PROCESS_NAME, RuleRelation.IS, "Autoruns.exe", RuleAction.EXCLUDE), Rule(Column.OPERATION, RuleRelation.BEGINS_WITH, "IRP_MJ_", RuleAction.EXCLUDE), Rule(Column.OPERATION, RuleRelation.BEGINS_WITH, "FASTIO_", RuleAction.EXCLUDE), Rule(Column.RESULT, RuleRelation.BEGINS_WITH, "FAST IO", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "pagefile.sys", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$Volume", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$UpCase", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$Secure", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$Root", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$MftMirr", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$Mft", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$LogFile", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.CONTAINS, "$Extend", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$Boot", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$Bitmap", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$BadClus", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$AttrDef", RuleAction.EXCLUDE), Rule(Column.EVENT_CLASS, RuleRelation.IS, "Profiling", RuleAction.EXCLUDE)]

Adding some new rules
>>> new_rules = [Rule('PID', 'is', '1336', 'include'), Rule('Process_Name', 'contains', 'python')]
>>> config["FilterRules"] = new_rules + config["FilterRules"]

Dropping filtered events
>>> config["DestructiveFilter"] = 1

Dumping the new configuration to a file
>>> with open("ProcmonConfiguration1337.pmc", "wb") as f:
... dump_configuration(config, f)

File Format
For the raw binary format of PMC files you can refer to the docs, or take a look at the source code in configuration_format.py.
PML (Process Monitor Log) Parser
Usage
procmon-parser exports a ProcmonLogsReader class for reading logs directly from a PML file:
>>> from procmon_parser import ProcmonLogsReader
>>> f = open("LogFile.PML", "rb")
>>> pml_reader = ProcmonLogsReader(f)
>>> len(pml_reader) # number of logs
53214

>>> first_event = next(pml_reader) # reading the next event in the log
>>> print(first_event)
Process Name=dwm.exe, Pid=932, Operation=RegQueryValue, Path="HKCU\Software\Microsoft\Windows\DWM\ColorPrevalence", Time=7/12/2020 1:18:10.7752429 AM

>>> print(first_event.process) # Accessing the process of the event
"C:\Windows\system32\dwm.exe", 932
>>> for module in first_event.process.modules[:3]:
... print(module) # printing information about some modules
"C:\Windows\system32\dwm.exe", address=0x7ff6fa980000, size=0x18000
"C:\Windows\system32\d3d10warp.dll", address=0x7fff96700000, size=0x76c000
"C:\Windows\system32\wuceffects.dll", address=0x7fff9a920000, size=0x3f000

>>> first_event.stacktrace # get a list of the stack frames addresses from the event
[18446735291098361031, 18446735291098336505, 18446735291095097155, 140736399934388, 140736346856333, 140736346854333, 140698742953668, 140736303659045, 140736303655429, 140736303639145, 140736303628747, 140736303625739, 140736303693867, 140736303347333, 140736303383760, 140736303385017, 140736398440420, 140736399723393]
>>>

File Format
For the raw binary format of PML files you can refer to the docs, or take a look at the source code in stream_logs_format.py.
Currently the parser is only tested with PML files saved by Procmon.exe of versions v3.4.0 or higher.
TODO
The PML format is very complex so there are some features (unchecked in the list) that are not supported yet:

Getting the IRP name of the operation.
Category column and Detail column, which contains different information about each operation type, is supported only for some of the operations:

Network operations

UDP/TCP Unknown
UDP/TCP Other
UDP/TCP Send
UDP/TCP Receive
UDP/TCP Accept
UDP/TCP Connect
UDP/TCP Disconnect
UDP/TCP Reconnect
UDP/TCP Retransmit
UDP/TCP TCPCopy


Process operations

Process Defined
Process Create
Process Exit
Thread Create
Thread Exit
Load Image
Thread Profile
Process Start
Process Statistics
System Statistics


Registry operations

RegOpenKey
RegCreateKey
RegCloseKey
RegQueryKey
RegSetValue
RegQueryValue
RegEnumValue
RegEnumKey
RegSetInfoKey
RegDeleteKey
RegDeleteValue
RegFlushKey
RegLoadKey
RegUnloadKey
RegRenameKey
RegQueryMultipleValueKey
RegSetKeySecurity
RegQueryKeySecurity


Filesystem Operations

VolumeDismount
VolumeMount
CreateFileMapping
CreateFile
CreatePipe
ReadFile
WriteFile
QueryInformationFile
SetInformationFile
QueryEAFile
SetEAFile
FlushBuffersFile
QueryVolumeInformation
SetVolumeInformation
DirectoryControl
FileSystemControl
DeviceIoControl
InternalDeviceIoControl
Shutdown
LockUnlockFile
CloseFile
CreateMailSlot
QuerySecurityFile
SetSecurityFile
Power
SystemControl
DeviceChange
QueryFileQuota
SetFileQuota
PlugAndPlay


Profiling Operations

Thread Profiling
Process Profiling
Debug Output Profiling





These are a lot of operation types so I didn't manage to get to all of them yet :(
If there is an unsupported operation which you think its details are interesting, please let me know :)
Tests
To test that the parsing is done correctly, There are two fairly large Procmon PML files and their respective CSV format
log files, taken from 64 bit and 32 bit machine. The test checks that each event in the PML parsed by procmon-parser
equals to the respective event in the CSV.
Contributing
procmon-parser is developed on GitHub at eronnen/procmon-parser.
Feel free to report an issue or send a pull request, use the
issue tracker.