GitLocker: The Coding Marketplace

Description:

kozmoserver 0.1.0

KozmoServer
An inference server for machine learning models.
Overview
KozmoServer aims to provide an easy way to start serving your machine learning
models through a REST and gRPC interface, fully compliant with KFServing's V2
Dataplane
spec. Watch a quick video introducing the project here.

Multi-model serving, letting users run multiple models within the same
process.
Ability to run inference in parallel for vertical
scaling
across multiple models through a pool of inference workers.
Support for adaptive
batching,
to group inference requests together on the fly.
Scalability with deployment in Kubernetes native frameworks, including
Kozmo Core and
KServe (formerly known as KFServing), where
KozmoServer is the core Python inference server used to serve machine learning
models.
Support for the standard V2 Inference Protocol on
both the gRPC and REST flavours, which has been standardised and adopted by
various model serving frameworks.

You can read more about the goals of this project on the initial design
document.
Usage
You can install the kozmoserver package running:
pip install kozmoserver

Note that to use any of the optional inference runtimes,
you'll need to install the relevant package.
For example, to serve a scikit-learn model, you would need to install the
kozmoserver-sklearn package:
pip install kozmoserver-sklearn

For further information on how to use KozmoServer, you can check any of the
available examples.
Inference Runtimes
Inference runtimes allow you to define how your model should be used within
KozmoServer.
You can think of them as the backend glue between KozmoServer and your machine
learning framework of choice.
You can read more about inference runtimes in their documentation
page.
Out of the box, KozmoServer comes with a set of pre-packaged runtimes which let
you interact with a subset of common frameworks.
This allows you to start serving models saved in these frameworks straight
away.
However, it's also possible to write custom
runtimes.
Out of the box, KozmoServer provides support for:

Framework
Supported
Documentation

Scikit-Learn
✅
KozmoServer SKLearn

XGBoost
✅
KozmoServer XGBoost

Spark MLlib
✅
KozmoServer MLlib

LightGBM
✅
KozmoServer LightGBM

CatBoost
✅
KozmoServer CatBoost

Tempo
✅
github.com/kozmoai/tempo

MLflow
✅
KozmoServer MLflow

Supervisor-Detect
✅
KozmoServer Supervisor Detect

Supervisor-Explain
✅
KozmoServer Supervisor Explain

HuggingFace
✅
KozmoServer HuggingFace

KozmoServer is licensed under the Apache License, Version 2.0. However please note that software used in conjunction with, or alongside, KozmoServer may be licensed under different terms. For example, Supervisor Detect and Supervisor Explain are both licensed under the Business Source License 1.1. For more information about the legal terms of products that are used in conjunction with or alongside KozmoServer, please refer to their respective documentation.
Supported Python Versions
🔴 Unsupported
🟠 Deprecated: To be removed in a future version
🟢 Supported
🔵 Untested

Python Version
Status

3.7
🔴

3.8
🔴

3.9
🟢

3.10
🟢

3.11
🔵

3.12
🔵

Examples
To see KozmoServer in action, check out our full list of
examples.
You can find below a few selected examples showcasing how you can leverage
KozmoServer to start serving your machine learning models.

Serving a scikit-learn model
Serving a xgboost model
Serving a lightgbm model
Serving a catboost model
Serving a tempo pipeline
Serving a custom model
Serving an supervisor-detect model
Serving a HuggingFace model
Multi-Model Serving with multiple frameworks
Loading / unloading models from a model repository

Developer Guide
Versioning
Both the main kozmoserver package and the inference runtimes
packages try to follow the same versioning schema.
To bump the version across all of them, you can use the
./hack/update-version.sh script.
We generally keep the version as a placeholder for an upcoming version.
For example:
./hack/update-version.sh 0.2.0.dev1

Testing
To run all of the tests for KozmoServer and the runtimes, use:
make test

To run run tests for a single file, use something like:
tox -e py3 -- tests/batch_processing/test_rest.py