GitLocker: The Coding Marketplace

Description:

juxtapose 0.0.35

🍿 Intro
Juxtapose is a 2D multi person pose detection, tracking, and estimation inference toolbox for sports + kinematics analysis. Visit Docs.

See how we integrated juxtapose into this app: Juxt Space
🍄 Overview
Code mostly adopted from four repos -> ultralytics, mmdeploy, mmdetection, mmpose.
Supported Detectors: rtmdet-s, rtmdet-m, rtmdet-l, groundingdino, yolov8
Supported Pose Estimators: rtmpose-s, rtmpose-m, rtmpose-l
Supported Trackers: bytetrack, botsort
Supported Point Trackers: Tapnet
🥒 Updates

2024/05/16 Remove ultralytics dependency, port yolov8 to run in ONNX directly to improve speed.
2024/04/27 Added FastAPI to EXE example with ONNX GPU Runtime in examples/fastapi-pyinstaller.
2024/01/11 Added Nextra docs + deployed to Vercel at sdk.juxt.space.
2024/01/07 Reduce dependencies by removing MMCV, MMDet, MMPose SDK, run fully on ONNX.
2023/11/01 Added juxtapose to PYPI repository so that we can install it using pip install juxtapose.
2023/08/25 Added custom region of interests (ROI) drawing tools that enables multi ROIs filtering while performing pose estimation/tracking. See usage below.
2023/08/15 Added GroundingDino & YOLOv8 object detector.
2023/08/09 Added keypoints streaming to csv file using csv module.
2023/07/31 Added ByteTrack and BotSORT. Completed engineering effort for top down inferences in any sources. See supported sources below.
2023/06/15 Converted RTMDET (s/m/l) and RTMPOSE (s/m/l) to ONNX using MMDeploy.

👉 Getting Started
Install Using PIP
pip install juxtapose
Note: If you faced any issues, kindly review this github issue
🧀 Local Development
git clone https://github.com/ziqinyeow/juxtapose
pip install .

🤩 Feel The Magic
🌄 Basic Usage
from juxtapose import RTM

# Init a rtm model (including rtmdet, rtmpose, tracker)
model = RTM(
det="rtmdet-m", # see type hinting
pose="rtmpose-m", # see type hinting
tracker="bytetrack", # see type hinting
device="cpu", # see type hinting
)

# Inference with directory (all the images and videos in the dir will get inference sequentially)
model("data")

# Inference with image
model("data/football.jpeg", verbose=False) # verbose -> disable terminal printing

# Inference with video
model("data/bike.mp4")

# Inference with the YouTube Source
model("https://www.youtube.com/watch?v=1vYvTbDJuFs&ab_channel=PeterGrant", save=True)

🎨 Select Region of Interests (ROIs)
It will first prompt the user to draw the ROIs, press r to remove the existing ROI drawn.
After drawing, press SPACE or ENTER or q to accept the ROI drawn. The model will filter
out the bounding boxes based on the ROIs.
😁 Note: Press SPACE again to redraw the bounding boxes. See custom implementation with cv2 here.
from juxtapose import RTM

model = RTM(det="groundingdino", pose="rtmpose-l", tracker="none")
model("data/bike.mp4", roi="rect") # rectangle roi

# 1. Draw ROI first
# 2. Press r or R to reset ROI
# 3. Press SPACE or Enter or q or Q to continue with the ROI

🚴‍♂️ Accessing result for each frame: More Flexibility
# Adding custom plot
import cv2
from juxtapose import RTM, Annotator

model = RTM()
annotator = Annotator(thickness=3, font_color=(128, 128, 128)) # see rtm.utils.plotting

# set show to true -> cv2.imshow the frame (you can use cv2 to plot anything in the frame)
# set plot to false -> if you want to ignore default plot -> see rtm.rtm (line `if plot:`)
for result in model("data/bike.mp4", show=True, plot=False, stream=True):
# do what ever you want with the data
im, bboxes, kpts = result.im, result.bboxes, result.kpts

# e.g custom plot anything using cv2 API
cv2.putText(
im, "custom text", (100, 100), cv2.FONT_HERSHEY_SIMPLEX, 1.0, (128, 128, 128)
)

# use the annotator class -> see rtm.utils.plotting
annotator.draw_bboxes(
im, bboxes, labels=[f"children_{i}" for i in range(len(bboxes))]
)
annotator.draw_kpts(im, kpts, thickness=4)
annotator.draw_skeletons(im, kpts)

⚽️ Custom Forward Pass: Full Flexibility
# Custom model forward pass
import cv2
import torch
from juxtapose import RTMDet, RTMPose, Annotator

frame = cv2.imread("data/football.jpeg")
device = "cuda" if torch.cuda.is_available() else "cpu"

# s, m, l
rtmdet = RTMDet("l", device=device)
rtmpose = RTMPose("l", device=device)
annotator = Annotator()

bboxes, scores, labels = rtmdet(frame) # [[x1, y1, x2, y2], ...], [], []
kpts = rtmpose(frame, bboxes=bboxes) # shape: (number of human, 17, 2)

annotator.draw_bboxes(frame, bboxes, labels=[f"person_{i}" for i in range(len(bboxes))])
annotator.draw_kpts(frame, kpts, thickness=4)
annotator.draw_skeletons(frame, kpts)

cv2.imshow("frame", frame)
cv2.waitKey(0)
cv2.destroyAllWindows()

Supported Sources
Adopted from ultralytics repository -> see https://docs.ultralytics.com/modes/predict/

Source
Argument
Type
Notes

image
'image.jpg'
str or Path
Single image file.

URL
'https://ultralytics.com/images/bus.jpg'
str
URL to an image.

screenshot
'screen'
str
Capture a screenshot.

PIL
Image.open('im.jpg')
PIL.Image
HWC format with RGB channels.

OpenCV
cv2.imread('im.jpg')
np.ndarray of uint8 (0-255)
HWC format with BGR channels.

numpy
np.zeros((640,1280,3))
np.ndarray of uint8 (0-255)
HWC format with BGR channels.

torch
torch.zeros(16,3,320,640)
torch.Tensor of float32 (0.0-1.0)
BCHW format with RGB channels.

CSV
'sources.csv'
str or Path
CSV file containing paths to images, videos, or directories.

video
'video.mp4'
str or Path
Video file in formats like MP4, AVI, etc.

directory
'path/'
str or Path
Path to a directory containing images or videos.

glob
'path/*.jpg'
str
Glob pattern to match multiple files. Use the * character as a wildcard.

YouTube
'https://youtu.be/Zgi9g1ksQHc'
str
URL to a YouTube video.

stream
'rtsp://example.com/media.mp4'
str
URL for streaming protocols such as RTSP, RTMP, or an IP address.