GitLocker: The Coding Marketplace

Description:

tritony 0.0.17

tritony - Tiny configuration for Triton Inference Server

What is this?
If you see the official example, it is really confusing to use where to start.
Use tritony! You will get really short lines of code like example below.
import argparse
import os
from glob import glob
import numpy as np
from PIL import Image

from tritony import InferenceClient

def preprocess(img, dtype=np.float32, h=224, w=224, scaling="INCEPTION"):
sample_img = img.convert("RGB")

resized_img = sample_img.resize((w, h), Image.Resampling.BILINEAR)
resized = np.array(resized_img)
if resized.ndim == 2:
resized = resized[:, :, np.newaxis]

scaled = (resized / 127.5) - 1
ordered = np.transpose(scaled, (2, 0, 1))

return ordered.astype(dtype)

if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--image_folder", type=str, help="Input folder.")
FLAGS = parser.parse_args()

client = InferenceClient.create_with("densenet_onnx", "0.0.0.0:8001", input_dims=3, protocol="grpc")
client.output_kwargs = {"class_count": 1}

image_data = []
for filename in glob(os.path.join(FLAGS.image_folder, "*")):
image_data.append(preprocess(Image.open(filename)))

result = client(np.asarray(image_data))

for output in result:
max_value, arg_max, class_name = output[0].decode("utf-8").split(":")
print(f"{max_value} ({arg_max}) = {class_name}")

Release Notes

24.07.11 Upgrade minimum tritonclient version to 2.34.0
23.08.30 Support optional with model input, parameters on config.pbtxt
23.06.16 Support tritonclient>=2.34.0
Loosely modified the requirements related to tritonclient

Key Features

Simple configuration. Only $host:$port and $model_name are required.
Generating asynchronous requests with asyncio.Queue
Simple Model switching
Support async tritonclient

Requirements
$ pip install tritonclient[all]

Install
$ pip install tritony

Test
With Triton
./bin/run_triton_tritony_sample.sh

pytest -s --cov-report term-missing --cov=tritony tests/

Example with image_client.py

Follow steps
in the official triton server documentation

# Download Images from https://github.com/triton-inference-server/server.git
python ./example/image_client.py --image_folder "./server/qa/images"