GitLocker: The Coding Marketplace

Description:

pgzip 0.3.5

pgzip

pgzip is a multi-threaded gzip implementation for python that increases the compression and decompression performance.
Compression and decompression performance gains are made by parallelizing the usage of block indexing within a gzip file. Block indexing utilizes gzip's FEXTRA feature which records the index of compressed members. FEXTRA is defined in the official gzip specification starting at version 4.3. Because FEXTRA is part of the gzip specification, pgzip is compatible with regular gzip files.
pgzip is ~25X faster for compression and ~7X faster for decompression when benchmarked on a 24 core machine. Performance is limited only by I/O and the python interpreter.
Theoretically, the compression and decompression speed should be linear with the number of cores available. However, I/O and a language's general performance limits the compression and decompression speed in practice.
Usage and Examples
CLI
❯ python -m pgzip -h
usage: __main__.py [-h] [-o OUTPUT] [-f FILENAME] [-d] [-l {0-9}] [-t THREADS] input

positional arguments:
input Input file or '-' for stdin

options:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
Output file or '-' for stdout (Default: Input file with 'gz' extension or stdout)
-f FILENAME, --filename FILENAME
Name for the original file when compressing
-d, --decompress Decompress instead of compress
-l {0-9}, --compression-level {0-9}
Compression level; 0 = no compression (Default: 9)
-t THREADS, --threads THREADS
Number of threads to use (Default: Determine automatically)

Programatically
Using pgzip is the same as using the built-in gzip module.
Compressing data and writing it to a file:
import pgzip

s = "a big string..."

# An explanation of parameters:
# `thread=8` - Use 8 threads to compress. `None` or `0` uses all cores (default)
# `blocksize=2*10**8` - Use a compression block size of 200MB
with pgzip.open("test.txt.gz", "wt", thread=8, blocksize=2*10**8) as fw:
fw.write(s)

Decompressing data from a file:
import pgzip

s = "a big string..."

with pgzip.open("test.txt.gz", "rt", thread=8) as fr:
assert fr.read(len(s)) == s

Performance
Compression Performance

Decompression Performance

Decompression was benchmarked using an 8.0GB FASTQ text file with 48 threads across 24 cores on a machine with Xeon(R) E5-2650 v4 @ 2.20GHz CPUs.
The compressed file used in this benchmark was created with a blocksize of 200MB.
Warning
pgzip only replaces the following methods of gzip's GzipFile class:

open()
compress()
decompress()

Other class methods and functionality have not been well tested.
Contributions or improvements is appreciated for methods such as:

seek()
tell()

History
Created initially by Vincent Li (@vinlyx), this project is a fork of https://github.com/vinlyx/mgzip. We had several bug fixes to implement, but we could not contact them. The pgzip team would like to thank Vincent Li (@vinlyx) for their hard work. We hope that they will contact us when they discover this project.