rocksq 0.3.0

Description:

rocksq 0.3.0

RocksQ
An inproc RocksDB-based queues with Python bindings.
The library is implemented in Rust and PyO3, which allows to release GIL when necessary. The library does not require
external dependencies to be installed in the environment.
Persistent queue
A persistent queue with following features:

max capacity limit in number of elements;
size calculation based on filesystem space usage;
length calculation based on number of elements;
supports only bytes-like objects;
can operate in a multithreaded environment efficiently (push and pop methods can release GIL if necessary);
keeps the state between restarts;
two implementations: blocking and nonblocking;

What is not supported:

pub/sub is not supported intentionally (implement it on top of RocksQ if necessary);
TTL is not supported intentionally (implement it on top of RocksQ if necessary).

Implementation details
It works on RocksDB and uses a single column family. The keys are 64-bit integers, the values are byte arrays. The keys
are generated by incrementing a counter. The read and write counters are stored in a separate key-value pairs.
MPMC queue
A persistent queue with following features:

TTL in seconds;
multiple consumers marked with labels;
size calculation based on filesystem space usage;
length calculation based on number of elements;
supports only bytes-like objects;
can operate in a multithreaded environment efficiently (add and next methods can release GIL if necessary);
keeps the state between restarts;
two implementations: blocking and nonblocking;

Implementation details
It works on RocksDB and uses three column families:


data
Stores queue elements. The keys are 64-bit integers, the values are byte arrays. The keys are generated by
incrementing a counter.


system
Stores a system information like start and write counters, a timestamp of the last write.


reader
Stores an information about consumers like read counters, expiration of elements after last reading. The keys are
string labels of consumers, the values are binary serialized objects.


TTL is implemented via RocksDB TTL feature. TTL is not strict.
It means that the element will remain in the queue for TTL seconds after insertion and the queue will make efforts to
remove the element after TTL seconds but it is not guaranteed to be done immediately. Thus, consumers can retrieve
expired but not removed elements.
Supported Platforms and Python Versions
Windows: Python versions: 3.7-3.12.
Linux: ManyLinux Python versions: 3.7-3.12. CI does not build for PyPy, but it should work if you build it manually.
MacOS: Currently, I do not have MacOS environment to debug the build process in MacOS, all volunteers are welcome.
Installation
pip install rocksq

Usage
See the examples in the python directory.
API docs are located at: https://insight-platform.github.io/RocksQ/.
Performance
The performance is mostly limited by the throughput of the underlying filesystem. The queue is able to saturate the
throughput of the filesystem.