arm-avhclient 0.1.10

Description:

armavhclient 0.1.10

Arm Virtual Hardware Client (avhclient)
The Arm Virtual Hardware Client (avhclient) is a python module that provides an interface for deploying and using Arm Virtual Hardware (AVH).
It enables uniform implementation of CI operations in various environments with reference examples provided for the following use cases:

Jenkins CI pipelines
GitHub-Actions workflows
Local use with AVH targets

Other environments can be supported using demonstrated concepts as well.
Example projects using AVH Client

AVH-GetStarted
CMSIS-RTOS2-Validation
TensorFlow Lite for Microcontrollers



Installation

Local installation
Installing (development snapshot) directly from GitHub main branch:
pip install git+https://github.com/ARM-software/avhclient.git@main


Docker container
Instead of installing Python and the AVH Client module into the local environment one
can use pre-built Docker images:
docker pull ghcr.io/arm-software/avhclient




Backend Setup
avhclient can control different backends with Arm Virtual Hardware. Following options are currently available:

aws (default) - interacts with AVH AMI available through AWS Marketplace
local - operates with AVH Targets installed locally.

The backend can be specified with -b option preceding the actual avhclient command.
Depending on the backend certain environment setup is expected.

AWS Backend Setup
avhclient accesses AWS services via Boto3 AWS SDK and for that requires a set of parameters to be available in the environment.

AWS Credentials
It is necessary to expose avhclient with the AWS credentials for your account.
You can either export your AWS credentials:
export AWS_ACCESS_KEY_ID="YOUR_AWS_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="YOUR_AWS_SECRET_ACCESS_KEY"
export AWS_SESSION_TOKEN="YOUR_AWS_SESSION_TOKEN"
or create a AWS credential file on ~/.aws/credentials (Linux & Mac) or %USERPROFILE%.awscredentials (Windows):
[default]
aws_access_key_id=YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key=YOUR_AWS_SECRET_ACCESS_KEY
aws_session_token=YOUR_AWS_SESSION_TOKEN
More info AWS CLI config and credentials


AWS Account info
In order for avhclient to create or access an AVH instance following parameters need to be defined in the execution environment of avhclient:


When creating and running a new AVH instance:
Mandatory info:
export AWS_IAM_PROFILE='YOUR_IAM_PROFILE'
export AWS_SECURITY_GROUP_ID='YOUR_AWS_SECURITY_GROUP_ID'
export AWS_SUBNET_ID='YOUR_SECURITY_GROUP_ID'
export AWS_S3_BUCKET_NAME='YOUR_B3_BUCKET_NAME'
Optional info (examples):
export AWS_AMI_ID=DESIRED_AVH_AMI_ID
export AWS_AMI_VERSION=1.1.2
export AWS_EFS_DNS_NAME=fs-066cf410af2428e2f.efs.eu-west-1.amazonaws.com
export AWS_EFS_PACKS_DIR=packs
export AWS_KEEP_EC2_INSTANCES=true
export AWS_KEY_NAME=YOUR_AWS_KEYPAIR_NAME
export AWS_INSTANCE_TYPE=t2.micro
export AWS_INSTANCE_NAME=MY_AVH_INSTANCE

If AWS_AMI_VERSION is not set, the avhclient will use the latest available version of AVH AMI.
If AWS_EFS_DNS_NAME is set, the AVH Client will try to mount it during the cloud-init phase. The only scenario supported for now is using Packs.
If AWS_EFS_PACKS_DIR is set, the mount path is relative to /home/ubuntu folder. Default folder is packs and if it exists locally will be then replaced by the EFS mount. Only used when AWS_EFS_DNS_NAME env is set.

AWS Cloudformation can be used to create the AWS resources required for AVH operation, as shown in this template



When reusing an AVH Instance:
export AWS_INSTANCE_ID=YOUR_INSTANCE





Local Backend Setup
Operation with a local backend requires no specific environment parameters, but assumes that necessary toolchain, AVH targets and utilities are installed locally on the machine and configured for execution in command line.




Usage

Getting Help
To get the brief descriptions of all commands and options available with avhclient execute:
avhclient -h
You can also use option -h with a specific command to get help for it. For example for execute command:
avhclient execute -h


Execute command


Create a new AWS AVH instance and run AVH project
The execute command bundles all necessary steps to build your
avh project:

prepare the backend.
upload your files
run your commands
download the results
cleanup the backend

Inform the path for the avh.yml file for your AVH project (example):
avhclient -b aws execute --specfile AVH-GetStarted/basic/avh.yml



You can also run in AVH commands in your local computer by selecting local backend:
avhclient -b local execute --specfile AVH-GetStarted/basic/avh.yml

There are also backend specific info you can provide to the tool:
avhclient -b aws -h (get full list)
avhclient -b aws --instance-name MY_NEW_NAME execute --specfile AVH-GetStarted/basic/avh.yml (seeting a new AVH instance name)
avhclient -b aws --ami-version 1.1.0 --specfile AVH-GetStarted/basic/avh.yml (Create a new AVH instance from a v1.1.0 AVH AMI)
avhclient -b aws --ami-version >1.1.0 --specfile AVH-GetStarted/basic/avh.yml (Create a new AVH instance from a >v1.1.0 AVH AMI)




Execute with Docker
To run avhclient in a Docker container one needs to create an environment file
(env.txt) with the following content:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_IAM_PROFILE
AWS_SECURITY_GROUP_ID
AWS_SUBNET_ID
AWS_S3_BUCKET_NAME
AWS_DEFAULT_REGION
AWS_AMI_ID
AWS_AMI_VERSION
AWS_KEEP_EC2_INSTANCES
AWS_KEY_NAME
AWS_INSTANCE_TYPE
AWS_INSTANCE_NAME
This environment file is used to forward the local environment variables into
the Docker container. Having this prepared one can run avhclient in a
container as follows:
docker run --rm -i --env-file ./env.txt \
-v $(pwd):/workspace \
-w /workspace \
ghcr.io/arm-software/avhclient \
avhclient [..]
The arguments are the same as above. If one requires more files from the Docker
host to be mapped into the container, this can be done like:
docker run --rm -i --env-file ./env.txt \
-v $HOME/.ssh:/root/.ssh \
-v $HOME/.aws:/root/.aws \
-v $(pwd):/workspace \
-w /workspace \
ghcr.io/arm-software/avhclient \
avhclient [..]
This exposes the local user’s SSH and AWS config files to the container.




AVH YML file syntax
avhclient execute command requires a specfile in YML format that describes details of individual steps to be executed on AVH. The file syntax is explained below.
A JSON schema for automatic checks and auto-completion is in schema/avh.schema.json.

Fields
Format of the specfile:
name: (optional) The name of the workload.
workdir: (optional) The local directory to use as the workspace, defaults to specfile's parent.
backend: (optional) Dictionary with backend specific parameters.
aws: (optional) Dictionary with AWS backend specific parameters. (see backend help)
local: (optional) Dictionary with local backend specific parameters. (see backend help)
upload: (optional) List of glob patterns of files (relative to workdir) to be sent to the AVH backend. (see glob format)
steps: (mandatory) List of steps to be executed on the AVH backend.
- run: String written into a bash script and executed on the AVH backend inside the workspace directory.
download: (optional) List of glob patterns of files (relative to workdir) to be retrieved back from the AVH backend. (see glob format)
Glob format:
The list of glob patterns is evaluated in order.
Wildcard '*' matches all files but no directory except hidden files (starting with '.').
Wildcard '**' matches all files and directories except hidden files/directories (starting with '.').
Inclusive matches (no prefix) are added to the file list.
Exclusive (prefixed with '-:') matches are removed from current file list.


Example
# yaml-language-server: $schema=https://raw.githubusercontent.com/ARM-software/avhclient/main/schema/avh.schema.json

name: "AVH GetStarted Example"
workdir: ./
backend:
aws:
ami-version: ~=1.1
instance-type: t2.micro
upload:
- RTE/**/*
- -:RTE/**/RTE_Components.h
- basic.debug.cprj
- build.py
- main.c
- requirements.txt
- retarget_stdio.c
- vht_config.txt
- README.md
steps:
- run: |
pip install -r requirements.txt
python build.py --verbose build run
download:
- RTE/**/RTE_Components.h
- Objects/basic.axf
- Objects/basic.axf.map
- basic-*.xunit
- basic-*.zip