jspcap 1.0.0

Description:

jspcap 1.0.0

jspcap

  The jspcap project is an open source Python program focus on PCAP parsing and analysis, which works as a stream PCAP file extractor. With support of jsformat, it shall support multiple output report formats.

Note that the whole project only supports Python 3.6 or later.


About

Module Structure

Foundation
Interface
Reassembly
IPSuite
Protocols
Utilities
CoreKit
DumpKit


Engine Comparison


Installation
Usage

Documentation

Interfaces
Macros

Formats
Layers
Engines


Protocols


CLI Usage


Samples

Usage Samples
CLI Samples


TODO


About
  jspcap is an independent open source library, using only jsformat as its formatted output dumper.

There is a project called jspcapy works on jspcap, which is a command line tool for PCAP extraction.

  Unlike popular PCAP file extractors, such as Scapy, dpkt, pyshark, and etc, jspcap uses streaming strategy to read input files. That is to read frame by frame, decrease occupation on memory, as well as enhance efficiency in some way.
Module Structure
  In jspcap, all files can be described as following six parts.

Foundation (jspcap.foundation) -- synthesise file I/O and protocol analysis, coordinate information exchange in all network layers
Interface (jspcap.interface) -- user interface for the jspcap library, which standardise and simplify the usage of this library
Reassembly (jspcap.reassembly) -- base on algorithms described in RFC 815, implement datagram reassembly of IP and TCP packets
IPSuite (jspcap.ipsuite) -- collection of constructors for Internet Protocol Suite
Protocols (jspcap.protocols) -- collection of all protocol family, with detail implementation and methods
Utilities (jspcap.utilities) -- collection of four utility functions and classes
CoreKit (jspcap.corekit) -- core utilities for jspcap implementation
DumpKit (jspcap.dumpkit) -- dump utilities for jspcap implementation


Engine Comparison
  Besides, due to complexity of jspcap, its extraction procedure takes around 0.01 seconds per packet, which is not ideal enough. Thus, jspcap introduced alternative extraction engines to accelerate this procedure. By now, jspcap supports Scapy, DPKT, and PyShark. Plus, jspcap supports two strategies of multiprocessing (server & pipeline). For more information, please refer to the document.



Engine
Performance (seconds per packet)




default
0.014525251388549805


server
0.12124489148457845


pipeline
0.014450424114863079


scapy
0.002443440357844035


dpkt
0.0003609057267506917


pyshark
0.0792640733718872



 
Installation

Note that jspcap only supports Python versions since 3.6

  Simply run the following to install the latest from PyPI:
pip install jspcap

  Or install from the git repository:
$ git clone https://github.com/JarryShaw/jspcap.git
$ python setup.py install

 
Usage
Documentation
Interfaces



NAME
DESCRIPTION




extract
extract a PCAP file


analyse
analyse application layer packets


reassemble
reassemble fragmented datagrams


trace
trace TCP packet flows



Macros
Formats



NAME
DESCRIPTION




JSON
JavaScript Object Notation (JSON) format


PLIST
macOS Property List (PLIST) format


TREE
Tree-View text format


PCAP
PCAP format



Layers



NAME
DESCRIPTION




RAW
no specific layer


LINK
data-link layer


INET
internet layer


TRANS
transport layer


APP
application layer



Engines



NAME
DESCRIPTION




jsPCAP
the default engine


MPServer
the multiprocessing engine with server process strategy


MPPipeline
the multiprocessing engine with pipeline strategy


DPKT
the DPKT engine


Scapy
the Scapy engine


PyShark
the PyShark engine



Protocols



NAME
DESCRIPTION




Raw
Raw Packet Data


ARP
Address Resolution Protocol


Ethernet
Ethernet Protocol


L2TP
Layer Two Tunnelling Protocol


OSPF
Open Shortest Path First


RARP
Reverse Address Resolution Protocol


VLAN
802.1Q Customer VLAN Tag Type


AH
Authentication Header


HIP
Host Identity Protocol


HOPOPT
IPv6 Hop-by-Hop Options


IP
Internet Protocol


IPsec
Internet Protocol Security


IPv4
Internet Protocol version 4


IPv6
Internet Protocol version 6


IPv6_Frag
Fragment Header for IPv6


IPv6_Opts
Destination Options for IPv6


IPv6_Route
Routing Header for IPv6


IPX
Internetwork Packet Exchange


MH
Mobility Header


TCP
Transmission Control Protocol


UDP
User Datagram Protocol


HTTP
Hypertext Transfer Protocol



  Documentation can be found in submodules of jspcap. Or, you may find usage sample in the test folder. For further information, please refer to the source code -- the docstrings should help you :)
ps: help function in Python should always help you out.
CLI Usage

The following part was originally described in jspcapy, which is now deprecated and merged into this repository.

  As it shows in the help manual, it is quite easy to use:
$ jspcapy --help
usage: jspcapy [-h] [-V] [-o file-name] [-f format] [-j] [-p] [-t] [-a] [-v]
[-F] [-E PKG] [-P PROTOCOL] [-L LAYER]
input-file-name

PCAP file extractor and formatted exporter

positional arguments:
input-file-name The name of input pcap file. If ".pcap" omits, it will
be automatically appended.

optional arguments:
-h, --help show this help message and exit
-V, --version show program's version number and exit
-o file-name, --output file-name
The name of input pcap file. If format extension
omits, it will be automatically appended.
-f format, --format format
Print a extraction report in the specified output
format. Available are all formats supported by
jsformat, e.g.: json, plist, and tree.
-j, --json Display extraction report as json. This will yield
"raw" output that may be used by external tools. This
option overrides all other options.
-p, --plist Display extraction report as macOS Property List
(plist). This will yield "raw" output that may be used
by external tools. This option overrides all other
options.
-t, --tree Display extraction report as tree view text. This will
yield "raw" output that may be used by external tools.
This option overrides all other options.
-a, --auto-extension If output file extension omits, append automatically.
-v, --verbose Show more information.
-F, --files Split each frame into different files.
-E PKG, --engine PKG Indicate extraction engine. Note that except default
or jspcap engine, all other engines need support of
corresponding packages.
-P PROTOCOL, --protocol PROTOCOL
Indicate extraction stops after which protocol.
-L LAYER, --layer LAYER
Indicate extract frames until which layer.

  Under most circumstances, you should indicate the name of input PCAP file (extension may omit) and at least, output format (json, plist, or tree). Once format unspecified, the name of output file must have proper extension (*.json, *.plist, or *.txt), otherwise FormatError will raise.
  As for verbose mode, detailed information will print while extraction (as following examples). And auto-extension flag works for the output file, to indicate whether extensions should be appended.
 
Samples
Usage Samples
  As described in test folder, jspcap is quite easy to use, with simply three verbs as its main interface. Several scenarios are shown as below.


extract a PCAP file and dump the result to a specific file (with no reassembly)
import jspcap
# dump to a PLIST file with no frame storage (property frame disabled)
plist = jspcap.extract(fin='in.pcap', fout='out.plist', format='plist', store=False)
# dump to a JSON file with no extension auto-complete
json = jspcap.extract(fin='in.cap', fout='out.json', format='json', extension=False)
# dump to a folder with each tree-view text file per frame
tree = jspcap.extract(fin='in.pcap', fout='out', format='tree', files=True)



extract a PCAP file and fetch IP packet (both IPv4 and IPv6) from a frame (with no output file)
>>> import jspcap
>>> extraction = jspcap.extract(fin='in.pcap', nofile=True)
>>> frame0 = extraction.frame[0]
# check if IP in this frame, otherwise ProtocolNotFound will be raised
>>> flag = jspcap.IP in frame0
>>> tcp = frame0[jspcap.IP] if flag else None



extract a PCAP file and reassemble TCP payload (with no output file nor frame storage)
import jspcap
# set strict to make sure full reassembly
extraction = jspcap.extract(fin='in.pcap', store=False, nofile=True, tcp=True, strict=True)
# print extracted packet if HTTP in reassembled payloads
for packet in extraction.reassembly.tcp:
for reassembly in packet.packets:
if jspcap.HTTP in reassembly.protochain:
print(reassembly.info)



CLI Samples
  The CLI (command line interface) of jspcap has two different access.

through console scripts -- use command name jspcapy directly (as shown in samples)
through Python module -- python -m jspcap [...] works exactly the same as above

Here are some usage samples:

export to a macOS Property List (Xcode has special support for this format)

$ jspcapy in --format plist --verbose
🚨Loading file 'in.pcap'
- Frame 1: Ethernet:IPv6:ICMPv6
- Frame 2: Ethernet:IPv6:ICMPv6
- Frame 3: Ethernet:IPv4:TCP
- Frame 4: Ethernet:IPv4:TCP
- Frame 5: Ethernet:IPv4:TCP
- Frame 6: Ethernet:IPv4:UDP
🍺Report file stored in 'out.plist'


export to a JSON file (with no format specified)

$ jspcapy in --output out.json --verbose
🚨Loading file 'in.pcap'
- Frame 1: Ethernet:IPv6:ICMPv6
- Frame 2: Ethernet:IPv6:ICMPv6
- Frame 3: Ethernet:IPv4:TCP
- Frame 4: Ethernet:IPv4:TCP
- Frame 5: Ethernet:IPv4:TCP
- Frame 6: Ethernet:IPv4:UDP
🍺Report file stored in 'out.json'


export to a text tree view file (without extension autocorrect)

$ jspcapy in --output out --format tree --verbose
🚨Loading file 'in.pcap'
- Frame 1: Ethernet:IPv6:ICMPv6
- Frame 2: Ethernet:IPv6:ICMPv6
- Frame 3: Ethernet:IPv4:TCP
- Frame 4: Ethernet:IPv4:TCP
- Frame 5: Ethernet:IPv4:TCP
- Frame 6: Ethernet:IPv4:UDP
🍺Report file stored in 'out'

 
TODO

specify Raw packet
interface verbs
review docstrings
merge jspcapy
write documentation
implement IP and MAC address containers
implement option list extractors
implement more protocols