GitLocker: The Coding Marketplace

Description:

datarules 0.2.0

DataRules
Goal and motivation
The idea of this project is to define rules to validate and correct datasets.
Whenever possible, it does this in a vectorized way, which makes this library fast.
Reasons to make this:

Implement an alternative to https://github.com/data-cleaning/ based on python and pandas.
Implement both validation and correction. Most existing packages provide validation only.
Support a rule based way of data processing. The rules can be maintained in a separate file (python or yaml) if required.
Apply vectorization to make processing fast.

Usage
This package provides two operations on data:

checks (if data is correct). Also knows as validations.
corrections (how to fix incorrect data)

Checks
In checks.py
from datarules import check

@check(tags=["P1"])
def check_almost_square(width, height):
return (width - height).abs() <= 4

@check(tags=["P3", "completeness"])
def check_not_too_deep(depth):
return depth <= 2

In your main code:
import pandas as pd
from datarules import CheckList

df = pd.DataFrame([
{"width": 3, "height": 7},
{"width": 3, "height": 5, "depth": 1},
{"width": 3, "height": 8},
{"width": 3, "height": 3},
{"width": 3, "height": -2, "depth": 4},
])

checks = CheckList.from_file('checks.py')
report = checks.run(df)
print(report)

Output:
name condition items passes fails NAs error warnings
0 check_almost_square check_almost_square(width, height) 5 3 2 0 None 0
1 check_not_too_deep check_not_too_deep(depth) 5 1 4 0 None 0

Corrections
In corrections.py
from datarules import correction
from checks import check_almost_square

@correction(condition=check_almost_square.fails)
def make_square(width, height):
return {"height": height + (width - height) / 2}

In your main code:
from datarules import CorrectionList

corrections = CorrectionList.from_file('corrections.py')
report = corrections.run(df)
print(report)

Output:
name condition action applied error warnings
0 make_square check_almost_square.fails(width, height) make_square(width, height) 2 None 0

Similar work (python)
These work on pandas, but only do validation:

Pandera - Like us, their checks are also vectorized.
Pandantic - Combination of validation and parsing based on pydantic.

The following offer validation only, but none of them seem to be vectorized or support pandas directly.

Great Expectations - An overengineered library for validation that has confusing documentation.
contessa - Meant to be used against databases.
validator
python-valid8
pyruler - Dead project that is rule-based.
pyrules - Dead project that supports rule based corrections (but no validation).

Similar work (R)
This project is inspired by https://github.com/data-cleaning/.
Similar functionality can be found in the following R packages:

validate - Checking data (implemented)
dcmodify - Correcting data (implemented)
errorlocate - Identifying and removing errors (not yet implemented)
deductive - Deductivate correction based on checks (not yet implemented)

Features found in one of the packages above but not implemented here, might eventually make it into this package too.