realign 0.1.38

Description:

realign 0.1.38

Realign: Evaluation & Experimentation Framework for AI Applications

realign is an evaluation and experimentation framework for building reliable AI applications through test-driven development. Test and evaluate agent architectures, RAG systems, prompts, and models across hundreds of scenarios specific to your use-case.
🎯 With Realign, you can:

Build reliable AI agents and RAG systems with test suites tailored to your use-case
Evaluate quality by simulating your agents over hundreds of scenarios in parallel
Experiment with 100+ models, prompts, and other parameters to find optimal configurations
Detect regressions by integrating test suites with your CI/CD pipeline
Track experiments with HoneyHive for cloud-scale analytics, visualization, and reproducibility

💡 What’s unique about Realign

YAML-Driven DX: Cleanly manage your agents, evaluator prompts, datasets, and other parameters using easy-to-read YAML config files
Composable Evaluators: Automatically evaluate quality using our library of 25+ pre-built evaluators, or create your own using composable building blocks
Blazing Fast Execution: Speed up your evaluations with parallel processing and async capabilities, with built-in modules for smart rate limiting
Statistical Rigor: Use statistics to test hypotheses and sweep hyperparameters to optimize performance

Quickstart
Installation & Setup
To install the package, run
pip install realign

Set your API keys as environment variables:
export OPENAI_API_KEY="your_openai_key"

or put them in a .env file:
OPENAI_API_KEY="your_openai_key"

Tweet Generator
Let's build an agent that produces content for your brand.

Tweetbot: generates N high quality tweets concurrently, evaluates them using Python code, HuggingFace classifiers, and LLM Judge (multiple runs, aggregated scores).

Please download and run this code!
Code: tweetbot.py
Config: config.yaml
Tutorials

Simple Tweet Bot: Generate tweets with any model using a prompt and template
Generate 10 Tweets in Parallel (Async): Generate tweets concurrently using async
Using Config Files: Setup config files to separate code and config
Set up Evaluators: Set up evaluator functions, new and built-in
Using Realign Evaluators: Use evaluators with configs

Concepts
1. @evaluator decorator
Learn how you can set up evaluators, and configure them with

wrapping
transforming
aggregating
checking
other settings and kwargs


An Evaluator is a function which scores your app's output and checks if the score is within a target range.

2. Simulation

A Simulation is a stochastic process that runs N times. It has statistical properties.


3. Agents

An LLM agent comprises the settings, instructions, and context given to an LLM to autonomously complete a certain task.

Set them up with


agent_name


the model settings


model: >100 providers/models


hyperparams: dictionary of OpenAI-type hyperparams




the prompt


system_prompt: a space for your agent's instructions


template: a template with variables marked with double curlies {{var}}


template_params: a dictionary mapping the variable names to their actual values


json_mode: a boolean flag which will deserialize the JSON response into a Python dict




Guides


[TODO] how do I evaluate my agent?


[TODO] how to I customize my evaluator?


[TODO] how do I improve my agent?


[TODO] how do I improve my RAG pipeline?


API Reference
coming soon!
Contributing
We welcome contributions from the community to help make Realign better. This guide will help you get started. If you have any questions, please reach out to us on Discord or through a GitHub issue.
Project Overview
Realign is an MIT licensed testing framework for multi-turn AI applications. It simulates user interactions, evaluates AI performance, and generates adversarial test cases.
We particularly welcome contributions in the following areas:


Bug fixes


Documentation updates, including examples and guides


Getting Started


Fork the repository on GitHub.


Clone your fork locally:


git clone https://github.com/[your-username]/realign.git

cd realign


Set up your development environment:

pip install -r requirements.txt

Development Workflow

Create a new branch for your feature or bug fix:

git checkout -b feature/your-feature-name



We try to follow the Conventional Commits specification. This is not required for feature branches. We merge all PRs into main with a squash merge and a conventional commit message.


Push your branch to your fork:


git push origin your-branch-name


Open a pull request against the main branch of the promptfoo repository.

When opening a pull request:


Keep changes small and focused. Avoid mixing refactors with new features.


Ensure test coverage for new code or bug fixes.


Provide clear instructions on how to reproduce the problem or test the new feature.


Be responsive to feedback and be prepared to make changes if requested.


Ensure your tests are passing and your code is properly linted.


Don't hesitate to ask for help. We're here to support you. If you're worried about whether your PR will be accepted, please talk to us first (see Getting Help).
Getting Help
If you need help or have questions, you can:


Open an issue on GitHub.


Join our Discord community.


Code of Conduct
We follow the Contributor Covenant Code of Conduct. Please read and adhere to it in all interactions within our community.