qalita 1.8.0

Description:

qalita 1.8.0

QALITA Command Line Interface (CLI)



QALITA Command Line Interface (CLI) is a tool intended to be used by Data Engineers who setup's QALITA Platform's agents, sources and assets.
It gives easy to use command to help them make an up & running qalita platform's environment in no time.

QALITA Command Line Interface (CLI)
Quick Start

Installation
Usage
Setup

Minimal Config
Connected Config
Make an .env file and export ENV Values:




qalita agent

qalita agent login
qalita agent run

Job
Worker


qalita agent joblist
qalita agent info


qalita pack

qalita pack init
qalita pack list
qalita pack run
qalita pack validate
qalita pack push


qalita source

qalita source add
qalita source validate
qalita source list
qalita source push


How to 'Pack' ?

Init
At runtime
Post runtime



Quick Start
Installation
As simple as :
pip install qalita

or there is a Docker Image available at :
Usage
If you want to have more detailed and contextual help, type
qalita COMMAND -h
Usage: qalita [OPTIONS] COMMAND [ARGS]...

QALITA Command Line Interface

Setup
This CLI command communicates with the QALITA Platform API backend.
There are several layers of configuration depending of your needs :
Minimal Config

QALITA_AGENT_NAME=<agent_name>

The agent will help you identify it in the frontend interface, there are no restrictions on the name.

QALITA_AGENT_MODE=<job/worker>

The mode of the agent :
Job : In job mode, when you use the command qalita agent run, it will immediately try to run a job on the local current context.
Worker : In worker mode, when you use the command qalita agent run it will wait for the backend to gives him jobs to run. It is simmilar to a scheduler.

Note that the command qalita agent run needs more configuration to run correctly, it will displays error otherwise.

Connected Config

QALITA_AGENT_ENDPOINT=<backend_api_url>

Example : http://localhost:3080/api/v1
The agent url endpoint gives the ability for the agent to communicate with the qalita's platform endpoints, it enables :
* Listing packs
* Running Jobs
* Publishing sources
* Publishing packs


QALITA_AGENT_TOKEN=<api_token>

The token is provided while doing the quickstart steps in the frontend app. It is associated with your user and your role.

Note that you need to have at least the [Data Engineer] role to use the QALITA CLI

Make an .env file and export ENV Values:
You can alternatively make an .env file and export the values to your environment.
.env-local
QALITA_AGENT_NAME=<agent_name>
QALITA_AGENT_MODE=<job/worker>
QALITA_AGENT_ENDPOINT=https://api.company.com/api/v1
QALITA_AGENT_TOKEN=<api_token>
QALITA_PACK_NAME=<pack_name>

Then export the values of the file to ENV values with :
export $(xargs < .env-local)

qalita agent
The qalita agent command allow you to :

Register an agent to the platform
Get information about your local agent
Run a pack on a source
List agent jobs (past & future)

qalita agent login
Parameters :

name : the name of the agent
mode : the mode of the agent <job/worker>
token : the api token you get from the platform
url : the backend api url of the platform

qalita agent login registers your local agent to the platform, it enables you to run jobs, or create routines (schedules) to run pack programmaticaly.
You need to have configured your agent with :

* QALITA_AGENT_ENDPOINT=<backend_api_url>
* QALITA_AGENT_TOKEN=<api_token>

You can get your token from the frontend or with an OAUTH2 API call to the /users/signin backend's endpoint
More info on your frontend documentation, and on the Connected config of the doc
qalita agent run
Parameters :

--name : the name of the agent
--mode : the mode of the agent <job/worker>
--token : the api token you get from the platform
--url : the backend api url of the platform

Specific parameters in job mode :

--source : the source id you want to run your job against
--source-version (optional) : the source version, by default it will run to the latest soruce version
--pack : the pack id you want to run your job against
--pack-version (optional) : the pack version, by default it will run the latest version of the pack

qalita agent run

runs in different mode :
Job
The agent will run given configuration

-p : a pack_id given with the qalita pack list, note that your pack needs to be pushed to the platform in order to have an id.
-s : a source_id given with the qalita source list, note that your source needs to be pushed to the platform in order to have an id.

Worker
The agent will wait until it receives an order from the frontend, it will then worke as same as in job mode.

Note that this mode will run indefinitely

qalita agent joblist
Parameters :

--name : the name of the agent
--mode : the mode of the agent <job/worker>
--token : the api token you get from the platform
--url : the backend api url of the platform

List jobs from the platform backend.
qalita agent info
Parameters :

--name : the name of the agent
--mode : the mode of the agent <job/worker>
--token : the api token you get from the platform
--url : the backend api url of the platform

Get infos about your local agent configuration.
qalita pack
The qalita pack command allow you to :

Initialize a new pack
List all available packs
Validate it
Run a local pack
Push your pack version to the platform

qalita pack init
Parameters :

--name : the name of the pack

Initialize a new pack, you need to have set a name, it will create a new folder with the name of the pack.
You can set your name by passing a new parameters to the commandline or setting a new environment variable : QALITA_PACK_NAME=<my-super-pack>.
Here is the arborescence created :
./<pack-name>_pack/
/run.sh # Entrypoint file that will be run with qalita agent run
/README.md # Documentation file
/properties.yaml # Properties file that contains properties about the pack
/main.py # (pack specific) The main script (you can run your pack with whatever langage you choose)
/config.json # (pack specific) The config file of your pack, you can use it to set any configurations you like.
/requirements.txt # (pack specific) The requirements file that is run inside the run.sh

qalita pack list
Parameters :

You need to have logged in with qalita agent login

List all the packs that are accessible to you with the Qalita Platform.
qalita pack run
Parameters :

--name : Pack name

Run your locally configured pack
qalita pack validate
Parameters :

--name : Pack name

Validate your locally configured pack
qalita pack push
Parameters :

--name : Pack name

Push your locally configured pack
qalita source
The qalita source command allow you to :

Add a new source to your local configuration
List your local sources from your qalita-conf.yml file
Push your local sources from your qalita-conf.yml file
Validate your conf file qalita-conf.yml


Note , by default the qalita-conf.yml file is stored to ~/.qalita/qalita-conf.yml , set QALITA_HOME env to customize the default path.

qalita source add
This function will help you add a new source to your configuration file qalita-conf.yaml
This command doesn't have parameters, you need to follow command prompts.

Prompt 1 : Source name
Prompt 2 : Source type
Prompt 3 : Is conditionnal, depends on the source type.

Case : Source Type = file : Source path
Case : Source Type = database : host / port / username / password / database


Prompt 4 : Source description
Prompt 5 : Is the source a reference ? [bool] (default : false)
Prompt 6 : Is the source sensistive ? [bool] (default : false)
Prompt 7 : Visibility of the source (private, internal, public) (default : private)

At the end of the prompt, the cli will check reachability of the source depending of the configuration and type, this step is called validate_source
to complete the process of registering a new source to the platform, you need to push your source with the command : qalita source push

qalita source validate
Helper function to help you add a new source to your configuration file qalita-conf.yaml
qalita source list
Parameters :
You need to have a qalita-conf.yaml file that contains your sources configuration.
Exemple :
version: 1
sources:
- config:
path: /home/user/data_dir
description: Folder containing csv files
name: my_csv_files
owner: user
reference: false
visibility: private
sensitive: false
type: file

In this exemple we have :
General keys



Key
Type
Description




version
int
The version of the configuration


sources
list
The list of sources



Source keys



Key
Type
Description




name
string
The name of the source


description
string
The description of the source


owner
string
The owner of the source


type
string
The type of the source


config
dict
The configuration of the source


visibility
string
The visibility of the source <private/internal/public>


reference
bool
Is the source a reference source


sensitive
bool
Is the source containing sensitive data



qalita source push
Registers your sources to the platform

Note: If you want to run a pack on your source, you will first need to push your source to the platform. It will give you a source_id with which you can run your pack.

How to 'Pack' ?
A pack is an entity run by the agent, it can be created by anyone.
It's purpose is to process the source and retrieve usefull informations about it to feed back into the platform.
Init
To create the base pack, see : qalita pack init.
At runtime
The entrypoint of the pack is the run.sh file that is located at the root path of the temp local folder created by the agent.
run.sh Example :
#/bin/bash
python -m pip install --quiet -r requirements.txt
python main.py

The pack is feed by a source_conf.json file containing the source's config: data. This file is located alongside the run.sh entrypoint.
source_conf.json Example :
{
"config": {
"path": "/home/lucas/desktop"
},
"description": "Desktop files",
"id": 1,
"name": "local_data",
"owner": "lucas",
"type": "file",
"reference": false,
"sensitive": false,
"visibility": "private",
"validate": "valid"
}


Note : The pack is responsible for managing itself it's source type compatibility by checking the source type in the source_conf.json file.

Post runtime
A the end of the pack run, the agent searchs for :

logs.txt : File uploaded to give feedback logs to the platform in the frontend.

logs.txt Example :
2023-07-21 11:51:12,688 - qalita.commands.pack - INFO - ------------- Pack Run -------------
2023-07-21 11:51:15,087 - qalita.commands.pack - INFO - CSV files found:
2023-07-21 11:51:15,222 - qalita.commands.pack - ERROR - Summarize dataset: 0%| | 0/5 [00:00<?, ?it/s]
...


recommendations.json

Recommendations file contains the recommendations given by the pack about the source.
recommendations.json Example :
{
[
{
"type":"<type of recommendation>",
"scope":"<scope : is a json>",
"content":"<any content>"
},
{
...
}
...
]
}


metrics.json

Metrics file contains the metrics given by the pack about the source.
metrics.json Example :
{
[
{
"scope":"<scope : is a json>",
"key":"<metric key>",
"value":"<metric value>"
},
{
...
}
...
]
}

Metrics & recommendations are pushed to the platform and are then available to the source's pack run view.