GitLocker: The Coding Marketplace

Description:

lokii 1.1.6

lokii is a powerful package that enables the generation of relational datasets, specifically tailored to
facilitate the creation of robust development environments. With lokii, you can effortlessly generate diverse
datasets that mimic real-world scenarios, allowing for comprehensive end-to-end testing of your applications.

Project structure
lokii leverages the hierarchical structure of the file system to discover groups and nodes. Each dataset
consists of nodes, which are defined using .node.py files. For instance, in the context of a database, each
node represents a table. Furthermore, you can even group nodes under database schemas within the database. Groups
defines how generated node data will be exported. You can recognize group files by their .group.py file extension.
# example project directory structure
proj_dir
├── group_1
│   ├── group_1.group.py
│   ├── node_2.node.py
│   └── node_2.node.py
├── group_2
│   ├── node_3.node.py
│   └── node_4.node.py
├── group_3.group.py
├── node_5.node.py
└── node_6.node.py

Node Definition
Node file defines how each item will be generated. There are special variables and functions in node
definition files.

name: Name of the node, filename will be used if not provided
source: Source query for retrieve dependent parameters for each item
item: Generation function that will return each item in node

# offices.node.py
from faker import Faker

# use your favorite tools to generate data
# you can even use database connection, filesystem or AI
fake = Faker()

# if you want you can override the node name if not provided filename will be used
# can be used in source queries if you want to retrieve rows that depends on another node
# name = "business.offices"

# define a query that returns one or more rows
source = "SELECT * FROM range(10)"

# item function will be called for each row in `source` query result
def item(args):
address = fake.address().split("\n")
return {
"officeCode": args["id"],
"city": fake.city(),
"phone": fake.phone_number(),
"addressLine1": address[0],
"addressLine2": address[1],
"state": fake.city(),
"country": fake.country(),
"postalCode": fake.postcode(),
"territory": fake.administrative_unit(),
}

Group Definition
Group file defines how each node data will be exported. There are special functions in group definition files.

before: Called once before export operation
export: Called for every node in the group
after: Called once after export operation

# filesystem.group.py
import os
import shutil
from csv import DictWriter

out_path = "out_data"

def before(args):
"""
Executed before export function.
:param args: contains node names that belongs to this group
:type args: {"nodes": list[str]}
"""
if os.path.exists(out_path):
# always clear your storage before starting a new export
shutil.rmtree(out_path)
os.makedirs(out_path)

def export(args):
"""
Executed for all nodes that belongs to this group
:param args: contains node name, node columns and a batch iterator
:type args: {"name": str, "cols": list[str], "batches": list[dict]}
"""
node_name = args["name"]
node_cols = args["cols"]
batches = args["batches"]
# out_data/offices.csv
out_file_path = os.path.join(out_path, node_name + ".csv")
with open(out_file_path, 'w+', newline='', encoding='utf-8') as outfile:
writer = DictWriter(outfile, fieldnames=node_cols)
writer.writeheader()
for batch in batches:
writer.writerows(batch)

def after(args):
"""
Executed after export function.
:param args: contains node names that belongs to this group
:type args: {"nodes": list[str]}
"""
pass

Upload to PyPI
You can create the source distribution of the package by running the command given below:
python3 setup.py sdist

Install twine and upload pypi for finnetdevlab username.
pip3 install twine
twine upload dist/*

Requirements
Package requirements are handled using pip. To install them do
pip install -r requirements.txt
pip install -r requirements.dev.txt

Tests
Testing is set up using pytest and coverage is handled
with the pytest-cov plugin.
Run your tests with py.test in the root directory.
Coverage is run by default and is set in the pytest.ini file.