GitLocker: The Coding Marketplace

Description:

lexy 0.0.3

Lexy
Lexy is a data platform for building AI-powered applications. You can use Lexy to:

Build Retrieval Augmented Generation (RAG) applications.
Provide relevant context for AI agents.
Add long-term memory to chat-based applications.
Extract structured data from unstructured documents.

To run Lexy locally, you'll need Docker installed. You'll also need Python 3.11
or greater. To install Lexy, you can either build from source (recommended) or install from PyPI (standalone mode).
Build from source
Clone the repo
git clone https://github.com/lexy-ai/lexy.git

Install dependencies
First create a virtual environment. Make sure that you're using Python 3.11 or greater. You can check your Python
version by running python3 --version.
# Change to the lexy directory
cd lexy
# Create a virtualenv
python3 -m venv venv
source venv/bin/activate

Next, install the dev dependencies. This will also create a .env file in the working directory if it doesn't
exist already.
# Install dev dependencies
make install-dev

Run docker compose
To build and run the docker containers, run the following command:
# Build and run docker containers
docker compose up --build -d

To stop the containers, run docker compose down.
Install from PyPI
Use this method if you're already familiar with Lexy and are using it as a dependency in your own project. You'll need
to provide instances of the Lexy server, worker, and database. See the Quickstart
guide for more information and a detailed example.
# Install server
pip install lexy[all]

# Install Python client
pip install lexy-py

# Initialize Lexy
lexy init

# Optional: Create a docker-compose file
lexy docker

Configuring storage
In order to upload and store files to Lexy, you'll need to configure either Amazon S3 or Google Cloud Storage.
Amazon S3
To configure Amazon S3, you can use aws configure on the command line (recommended) or set the value of
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your .env file.
If you choose to use aws configure, make sure to uncomment the following line in docker-compose.yaml
in order for your credentials to be available inside of the lexyserver container.
volumes:
# Uncomment the following line to mount local AWS credentials
- $HOME/.aws/credentials:/root/.aws/credentials:ro

You'll also need to specify an S3 bucket for file storage (for which your AWS credentials should have full access).
You can do so by adding DEFAULT_STORAGE_BUCKET=<name-of-your-S3-bucket> to your .env file. Remember to rebuild
your dev containers for the change to take effect (run make rebuild-dev-containers on the command line).
# Example .env settings for Amazon S3
DEFAULT_STORAGE_SERVICE=s3
DEFAULT_STORAGE_BUCKET=your_s3_bucket_name
AWS_ACCESS_KEY_ID=your_aws_access_key_id
AWS_SECRET_ACCESS_KEY=your_aws_secret_access_key

Google Cloud Storage
To use Google Cloud Storage, you'll need to set DEFAULT_STORAGE_SERVICE=gcs in your .env file. You'll also need to
create a service account, and set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of your service
account credentials. You can do so by adding the following to your .env file:
GOOGLE_APPLICATION_CREDENTIALS=<path-to-your-service-account-credentials>

You'll also need to specify a GCS bucket for file storage (for which your service account should have full access).
You can do so by adding DEFAULT_STORAGE_BUCKET=<name-of-your-GCS-bucket> to your .env file. Remember to rebuild
your dev containers for the change to take effect (run make rebuild-dev-containers on the command line).
# Example .env settings for Google Cloud Storage
DEFAULT_STORAGE_SERVICE=gcs
DEFAULT_STORAGE_BUCKET=your_gcs_bucket_name
GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/service-account-credentials.json

Using OpenAI transformers
To use OpenAI embeddings in Lexy, you'll need to set the OPENAI_API_KEY environment variable. You can do so by adding
the following to your .env file:
OPENAI_API_KEY=<your-openai-api-key>

Do this before building your docker containers. Or, if you've already run docker compose up, you can run the
following to rebuild the server and worker containers.
# Rebuild the server and worker containers
make rebuild-dev-containers

Lexy dashboard (WIP)
Lexy comes with a built-in dashboard to visualize pipelines. This is still under development, but you can run it locally.
See the dashboard README for more details.
To start the dashboard, make sure you have Node.js installed. Then, from the root directory, run the following commands:
cd dashboard
npm install
npm run dev

The dashboard will be running at http://localhost:3000.
Where to find services
The server will be running at http://localhost:9900. In addition, you can find the following services.

Service
URL
Notes

Lexy API
http://localhost:9900/docs
Swagger API docs

Flower
http://localhost:5556
Celery task monitor

RabbitMQ
http://localhost:15672
Username: guest, Password: guest

Postgres
http://localhost:5432
Database: lexy, Username: postgres, Password: postgres

Project docs
http://localhost:8000
Run make serve-docsUsername: lexy, Password: guest

Lexy dashboard
http://localhost:3000
Dashboard to show pipelines (WIP)

Troubleshooting
PyCharm issues
If your virtualenv keeps getting bjorked by PyCharm, make sure that you're following the instructions above verbatim,
and using venv instead of .venv for the path of your virtual environment.
Other issues
If you encounter any issues while installing or using Lexy, please open an
issue on GitHub.