py-scdb 0.2.2

Description:

pyscdb 0.2.2

py_scdb

A very simple and fast key-value (as UTF-8 strings) store but persisting data to disk, with a "localStorage-like" API.
This is the python version of the original scdb
scdb may not be production-ready yet. It works, quite well but it requires more rigorous testing.
Purpose
Coming from front-end web
development, localStorage was always
a convenient way of quickly persisting data to be used later by a given application even after a restart.
Its API was extremely simple i.e. localStorage.getItem(), localStorage.setItem(), localStorage.removeItem()
, localStorage.clear().
Coming to the backend (or even desktop) development, such an embedded persistent data store with a simple API
was hard to come by.
scdb is meant to be like the 'localStorage' of backend and desktop (and possibly mobile) systems.
Of course to make it a little more appealing, it has some extra features like:

Time-to-live (TTL) where a key-value pair expires after a given time
Non-blocking reads from separate processes, and threads.
Fast Sequential writes to the store, queueing any writes from multiple processes and threads.
Optional searching of keys that begin with a given subsequence. This option is turned on when scdb::new() is called.
Note: When searching is enabled, delete, get, compact, clear become considerably slower.

Dependencies

python +v3.7

Quick Start (Synchronous)

Install the package

pip install py_scdb


Import the Store and use accordingly

if __name__ == "__main__":
from py_scdb import Store

# These need to be UTF-8 strings
records = [
("hey", "English"),
("hi", "English"),
("salut", "French"),
("bonjour", "French"),
("hola", "Spanish"),
("oi", "Portuguese"),
("mulimuta", "Runyoro"),
]
keys = [k for (k, _) in records]

store = Store(
store_path="db",
max_keys=1000000,
redundant_blocks=1,
pool_capacity=10,
compaction_interval=1800,
is_search_enabled=True,
)

# inserting without ttl
for (k, v) in records[:3]:
store.set(k=k, v=v)

# inserting with ttl of 5 seconds
for (k, v) in records[3:]:
store.set(k=k, v=v, ttl=5)

# updating - just set them again
updates = [
("hey", "Jane"),
("hi", "John"),
("hola", "Santos"),
("oi", "Ronaldo"),
("mulimuta", "Aliguma"),
]
for (k, v) in updates:
store.set(k=k, v=v)

# getting
for k in keys:
v = store.get(k=k)
print(f"Key: {k}, Value: {v}")

# searching without pagination
results = store.search(term="h")
print(f"Search 'h' (no pagination):\n{results}\n")

# searching with pagination
results = store.search(term="h", skip=1, limit=2)
print(f"Search 'h' (skip=1, limit=2):\n{results}\n")

# deleting
for k in keys[:3]:
store.delete(k=k)

# clearing
store.clear()

# compacting (Use sparingly, say if database file is too big)
store.compact()


Run the module

python <module_name>.py
# e.g. python main.py

Quick Start (Asynchronous)

Install the package

pip install py_scdb


Import the AsyncStore and use accordingly

import asyncio
from py_scdb import AsyncStore

# These need to be UTF-8 strings
records = [
("hey", "English"),
("hi", "English"),
("salut", "French"),
("bonjour", "French"),
("hola", "Spanish"),
("oi", "Portuguese"),
("mulimuta", "Runyoro"),
]
keys = [k for (k, _) in records]

async def run_async_example():
store = AsyncStore(
store_path="db",
max_keys=1000000,
redundant_blocks=1,
pool_capacity=10,
compaction_interval=1800,
is_search_enabled=True,
)

# inserting without ttl
for (k, v) in records[:3]:
await store.set(k=k, v=v)

# inserting with ttl of 5 seconds
for (k, v) in records[3:]:
await store.set(k=k, v=v, ttl=5)

# updating - just set them again
updates = [
("hey", "Jane"),
("hi", "John"),
("hola", "Santos"),
("oi", "Ronaldo"),
("mulimuta", "Aliguma"),
]
for (k, v) in updates:
await store.set(k=k, v=v)

# getting
for k in keys:
v = await store.get(k=k)
print(f"Key: {k}, Value: {v}")

# searching without pagination
results = await store.search(term="h")
print(f"Search 'h' (no pagination):\n{results}\n")

# searching with pagination
results = await store.search(term="h", skip=1, limit=2)
print(f"Search 'h' (skip=1, limit=2):\n{results}\n")

# deleting
for k in keys[:3]:
await store.delete(k=k)

# clearing
await store.clear()

# compacting (Use sparingly, say if database file is too big)
await store.compact()


asyncio.run(run_async_example())


Run the module

python <module_name>.py
# e.g. python main.py

Contributing
Contributions are welcome. The docs have to maintained, the code has to be made cleaner, more idiomatic and faster,
and there might be need for someone else to take over this repo in case I move on to other things. It happens!
Please look at the CONTRIBUTIONS GUIDELINES
You can also look in the ./docs
folder of the rust scdb to get up to speed with the internals of scdb e.g.

database file format
how it works
inverted index file format
how the search works

Bindings
scdb is meant to be used in multiple languages of choice. However, the bindings for most of them are yet to be
developed.
For other programming languages, see the main README
How to Test

Clone the repo and enter its root folder

git clone https://github.com/sopherapps/py_scdb.git && cd py_scdb


Create a virtual environment and activate it

virtualenv -p /usr/bin/python3.7 env && source env/bin/activate


Install the dependencies

pip install -r requirements.txt


Install scdb package in the virtual environment

maturin develop

For optimized build use:
maturin develop -r


Run the tests command

pytest --benchmark-disable


Run benchmarks

pytest --benchmark-compare --benchmark-autosave

OR the summary
# synchronous API
pytest test/test_benchmarks.py --benchmark-columns=mean,min,max --benchmark-name=short --benchmark-sort=NAME

Benchmarks
On an average PC (17Core, 16 GB RAM)
Synchronous

--------------------------------------------------------- benchmark: 40 tests ---------------------------------------------------------
Name (time in us) Mean Min Max
---------------------------------------------------------------------------------------------------------------------------------------
benchmark_clear[sync_store] 100.3807 (127.70) 83.0300 (116.78) 311.0190 (6.71)
benchmark_clear_with_search[sync_searchable_store] 171.0275 (217.57) 133.6630 (187.99) 433.9330 (9.36)
benchmark_compact[sync_store] 110,236.5678 (>1000.0) 106,076.8240 (>1000.0) 117,694.6450 (>1000.0)
benchmark_compact_with_search[sync_searchable_store] 111,180.5895 (>1000.0) 102,213.0760 (>1000.0) 127,501.7640 (>1000.0)
benchmark_delete[sync_store-hey] 3.7683 (4.79) 3.4310 (4.83) 82.3730 (1.78)
benchmark_delete[sync_store-hi] 3.7664 (4.79) 3.4440 (4.84) 54.3430 (1.17)
benchmark_delete_with_search[sync_searchable_store-hey] 20.7839 (26.44) 15.0590 (21.18) 122.8020 (2.65)
benchmark_delete_with_search[sync_searchable_store-hi] 20.7891 (26.45) 14.7560 (20.75) 123.6690 (2.67)
benchmark_get[sync_store-hey] 0.7861 (1.0) 0.7110 (1.0) 64.0260 (1.38)
benchmark_get[sync_store-hi] 0.7877 (1.00) 0.7190 (1.01) 55.0760 (1.19)
benchmark_get_with_search[sync_searchable_store-hey] 0.7991 (1.02) 0.7220 (1.02) 49.2670 (1.06)
benchmark_get_with_search[sync_searchable_store-hi] 0.7899 (1.00) 0.7230 (1.02) 46.3370 (1.0)
benchmark_paginated_search[sync_searchable_store-b] 12.9258 (16.44) 12.1430 (17.08) 156.1050 (3.37)
benchmark_paginated_search[sync_searchable_store-ba] 12.9307 (16.45) 12.1640 (17.11) 115.5400 (2.49)
benchmark_paginated_search[sync_searchable_store-ban] 7.0885 (9.02) 6.6530 (9.36) 71.0410 (1.53)
benchmark_paginated_search[sync_searchable_store-bar] 6.9927 (8.90) 6.6160 (9.31) 82.8730 (1.79)
benchmark_paginated_search[sync_searchable_store-f] 13.2767 (16.89) 12.0420 (16.94) 21,129.0600 (455.99)
benchmark_paginated_search[sync_searchable_store-fo] 12.9289 (16.45) 12.1860 (17.14) 439.6100 (9.49)
benchmark_paginated_search[sync_searchable_store-foo] 12.9970 (16.53) 12.3170 (17.32) 102.5070 (2.21)
benchmark_paginated_search[sync_searchable_store-for] 6.9771 (8.88) 6.6010 (9.28) 94.3090 (2.04)
benchmark_paginated_search[sync_searchable_store-p] 6.9895 (8.89) 6.5870 (9.26) 149.4620 (3.23)
benchmark_paginated_search[sync_searchable_store-pi] 6.9171 (8.80) 6.6080 (9.29) 69.2660 (1.49)
benchmark_paginated_search[sync_searchable_store-pig] 6.9475 (8.84) 6.5980 (9.28) 67.4790 (1.46)
benchmark_paginated_search[sync_searchable_store-pigg] 6.9717 (8.87) 6.6070 (9.29) 111.5230 (2.41)
benchmark_search[sync_searchable_store-b] 15.5810 (19.82) 14.6940 (20.67) 105.9520 (2.29)
benchmark_search[sync_searchable_store-ba] 15.6253 (19.88) 14.7780 (20.78) 107.5030 (2.32)
benchmark_search[sync_searchable_store-ban] 10.5073 (13.37) 9.9340 (13.97) 118.7900 (2.56)
benchmark_search[sync_searchable_store-bar] 10.8698 (13.83) 9.8550 (13.86) 97.3030 (2.10)
benchmark_search[sync_searchable_store-f] 20.7594 (26.41) 19.5820 (27.54) 116.9510 (2.52)
benchmark_search[sync_searchable_store-fo] 20.8279 (26.50) 19.6790 (27.68) 196.4210 (4.24)
benchmark_search[sync_searchable_store-foo] 15.7794 (20.07) 14.8380 (20.87) 94.2340 (2.03)
benchmark_search[sync_searchable_store-for] 10.4470 (13.29) 9.8640 (13.87) 128.4710 (2.77)
benchmark_search[sync_searchable_store-p] 10.3812 (13.21) 9.7870 (13.77) 102.3630 (2.21)
benchmark_search[sync_searchable_store-pi] 10.3460 (13.16) 9.8380 (13.84) 84.1260 (1.82)
benchmark_search[sync_searchable_store-pig] 10.4069 (13.24) 9.8400 (13.84) 78.9080 (1.70)
benchmark_search[sync_searchable_store-pigg] 6.9492 (8.84) 6.6000 (9.28) 86.0650 (1.86)
benchmark_set[sync_store-hey-English] 10.6412 (13.54) 9.2990 (13.08) 110.4370 (2.38)
benchmark_set[sync_store-hi-English] 10.7570 (13.68) 9.1760 (12.91) 100.9780 (2.18)
benchmark_set_with_search[sync_searchable_store-hey-English] 37.2746 (47.42) 32.7920 (46.12) 8,647.2350 (186.62)
benchmark_set_with_search[sync_searchable_store-hi-English] 28.0105 (35.63) 25.5050 (35.87) 105.3320 (2.27)
---------------------------------------------------------------------------------------------------------------------------------------

Acknowledgement

The asyncio was got using pyo3-asyncio
The python-rust bindings were got from the pyo3 project

License
Licensed under both the MIT License and the APACHE (2.0) License
Copyright (c) 2022 Martin Ahindura
Copyright (c) 2017-present PyO3 Project and Contributors
Gratitude

"Come to Me (Christ Jesus), all you who labor and are heavy laden, and I will give you rest.
Take My yoke upon you and learn from Me, for I am gentle and lowly in heart,
and you will find rest for your souls. For My yoke is easy and My burden is light."
-- Matthew 11: 28-30

All glory be to God.