GitLocker: The Coding Marketplace

Description:

oreum core 0.8.1

Oreum Core Tools oreum_core

1. Description and Scope
This is an ever-growing package of core tools for use on client projects by
Oreum Industries.

Provides an essential workflow for data curation, EDA, basic ML using the core
scientific Python stack incl. numpy, scipy, matplotlib, seaborn,
pandas, scikit-learn, umap-learn
Optionally provides an advanced Bayesian modelling workflow in R&D and
Production using a leading probabilistic programming stack incl. pymc,
pytensor, arviz
(do pip install oreum_core[pymc])
Optionally enables a generalist black-box ML workflow in R&D using a leading
Gradient Boosted Trees stack incl. catboost, xgboost, optuna, shap
(do pip install oreum_core[tree])
Also includes several utilities for text cleaning, sql scripting, file handling

This package is:

A work in progress (v0.y.z) and liable to breaking changes and inconvenience
to the user
Solely designed for ease of use and rapid development by employees of
Oreum Industries, and selected clients with guidance

This package is not:

Intended for public usage and will not be supported for public usage
Intended for contributions by anyone not an employee of Oreum Industries,
and unsolicitied contributions will not be accepted.

Notes

Project began on 2021-01-01
The README.md is MacOS and POSIX oriented
See LICENCE.md for licensing and copyright details
See pyproject.toml for various pacakge details
This uses a logger named 'oreum_core', feel free to incorporate or ignore
Hosting:

Source code repo on GitHub
Source code release on GitHub
Package release on PyPi

2. Instructions to Create Dev Environment
For local development on MacOS
2.0 Pre-requisite installs via homebrew

Install Homebrew, see instuctions at https://brew.sh
Install direnv, git, git-lfs, graphviz, zsh

$> brew update && upgrade
$> brew install direnv git git-lfs graphviz zsh

2.1 Git clone the repo
Assumes direnv, git, git-lfs and zsh installed as above
$> git clone https://github.com/oreum-industries/oreum_core
$> cd oreum_core

Then allow direnv on MacOS to autorun file .envrc upon directory open
2.2 Create virtual environment and install dev packages
Notes:

We use conda virtual envs controlled by mamba (quicker than conda)
We install packages using miniforge (sourced from the conda-forge repo)
wherever possible and only use pip for packages that are handled better by
pip and/or more up-to-date on pypi
Packages might not be the very latest because we want stability for pymc
which is usually in a state of development flux
See cheat sheet of conda commands
The Makefile creates a dev env and will also download and preinstall
miniforge if not yet installed on your system

2.2.1 Create the dev environment
From the dir above oreum_core/ project dir:
$> make -C oreum_core/ dev

This will also create some files to help confirm / diagnose successful installation:

dev/install_log/blas_info.txt for the BLAS MKL installation for numpy
dev/install_log/pipdeptree[_rev].txt lists installed package deps (and reversed)
LICENSES_THIRD_PARTY.md details the license for each package used

2.2.2 (Optional best practice) Test successful installation of dev environment
From the dir above oreum_core/ project dir:
$> make -C oreum_core/ test-dev-env

This will also add files dev/install_log/[numpy|scipy].txt which detail
successful installation (or not) for numpy, scipy
2.2.3 (Useful during env install experimentation): To remove the dev environment
From the dir above oreum_core/ project dir:
$> make -C oreum_core/ uninstall-env

2.3 Code Linting & Repo Control
2.3.1 Pre-commit
We use pre-commit to run a suite of automated tests
for code linting & quality control and repo control prior to commit on local
development machines.

Precommit is already installed by the make dev command (which itself calls
pip install -e .[dev])
The pre-commit script will then run on your system upon git commit
See this project's .pre-commit-config.yaml for details

2.3.2 Github Actions
We use Github Actions aka
Github Workflows to run:

A suite of automated tests for commits received at the origin (i.e. GitHub)
Publishing to PyPi upon creating a GH Release

See Makefile for the CLI commands that are issued
See .github/workflows/* for workflow details

Copyright 2024 Oreum OÜ t/a Oreum Industries. All rights reserved.
See LICENSE.md.
Oreum OÜ t/a Oreum Industries, Sepapaja 6, Tallinn, 15551, Estonia,
reg.16122291, oreum.io

Oreum OÜ © 2024