Description:
polarstdigest 0.1.3
T-Digest Polars Plugin
This Polars plugin is a wrapper around the T-Digest Rust implementation. It not only provides means to compute an estimated qunatile, but exposes the tdigest creation and merging functionality so it can be used to estimate quantiles in a distributed environment.
For an example see the Yellow Taxi Notebook. Note that this example is a bit artifical as it doesn't distribute the computation. It is mainly meant to show how to use the plugin with multiple partitions of a dataset. It does not make sense to use this plugin for computations on a single machine as the tdigest computation essentially adds overhead to the percentile computation and is therefore slower than computing the actual percentile.
How to contribute
Dev setup
Setup your virtual environment with a python version >=3.8, e.g. use python -m venv .env.
Install the python dependencies used for development:
python -m pip install -r requirements.txt
Install Rust.
Build
In order to build the package, please run maturin develop. If you want to test performance, run maturin develop --release.
Commit / Release
Before committing and pushing your work, make sure to run
cargo fmt --all && cargo clippy --all-features
python -m ruff check . --fix --exit-non-zero-on-fix
and resolve any errors.
License
For personal and professional use. You cannot resell or redistribute these repositories in their original state.
Customer Reviews
There are no reviews.
