GitLocker: The Coding Marketplace

Description:

pipegen 0.5.0

pipegen

A CodePipeline/CodeBuild CI/CD Pipeline creator + deployer.
Installation
Install using pip/pypi:
pip install pipegen

Usage
Deploying your pipeline with pipegen:
pipegen deploy --config CONFIG_FILE --stack-name NAME_OF_STACK [--var KEY=VALUE [--var KEY=VALUE]]

To output compiled configuration:
pipegen dump config --config CONFIG_FILE [--var KEY=VALUE [--var KEY=VALUE]]

To output compiled CloudFormation template:
pipegen dump template --config CONFIG_FILE [--var KEY=VALUE [--var KEY=VALUE]]

Configuration Schema
The schema is broken down into several sections:

Base/shared config used across many of the resources
Source configuration
Pipeline configuration

Base Config
Base config is used across multiple resources deployed using pipegen, or can be used to override some defaults.
Full schema is documented below, required elements are specified with (R), and defaults are indicated.
config:
s3_bucket: (R) the name of a S3 bucket to store artifacts in (source copies,
codebuild artifacts, etc)
kms_key_arn: (R) the ARN of a KMS key to encrypt all resources with

codepipeline:
restart_execution_on_update: whether or not to restart an in-progress
CodePipeline execution if the pipeline is
updated (default: false)

codebuild:
compute_type: the default compute type to use (default: BUILD_GENERAL1_SMALL)
image: the default docker image to use
(default: aws/codebuild/amazonlinux2-x86_64-standard:3.0)
log_group:
enabled: whether or not to enable CloudWatch logs for CodeBuild
projects (default: true)
create: whether or not to create the CloudWatch log group (default: true).
Disable if you have created the log group externally.
name: the name of the CloudWatch log group for the pipeline
(default: random generated name based on the stack name)
retention: A number in days to retain logs (default: null, logs are retained
indefinitely)
iam: a list of IAM statements to add to the CodeBuild role (default: null).
Use if your CodeBuild projects need to manipulate AWS resources

IAM Examples
By default, pipegen configures CodeBuild with the minimal amount of permissions in order to run, decrypt your artifacts from KMS, pull images from ECR (if configured), write logs to CloudWatch logs (if configured). If you require additional IAM permissions, you can specify them using the following syntax:
config:
iam:
- standard IAM statement

For example, adding permissions to upload files to S3 in your build:
config:
iam:
- Effect: Allow
Action:
- s3:PutObject
Resource:
- my-bucket-name/*

Source configuration
Source configuration defines where your project's code comes from.
Two top-level options are supported currently:

CodeCommit, AWS' hosted git service
CodeStar Connections, to use Github.com, GitHub Enterprise (GHE), or BitBucket

Pipelines can use multiple source repositories mixed from either provider.
CodeCommit
The following describes source configuration for using CodeCommit.
Full schema is documented below, required elements are specified with (R), and defaults are indicated.
sources:
- name: (R) a unique name for this source entry
from: (R) the provider to use. use `CodeCommit` for CodeCommit
repository: (R) the name of the repository
branch: (R) the branch to build from
poll_for_source_changes: whether CodePipeline should poll for updates (default: false)
event_for_source_changes: whether to use CloudWatch events/EventBridge to
detect updates (default: true)

Note: if you want the pipeline to trigger a build when the repository is pushed to, either poll_for_source_changes or event_for_source_changes should be set to true (event_for_source_changes is preferred).
CodeStar Connections
The following describes source configuration for using CodeStar Connections to get your source from Github.com, GHE or BitBucket.
In order to use CodeStar connections, you must have configurared a connection separately and prior to deploying the pipeline stack.
Full schema is documented below, required elements are specified with (R), and defaults are indicated.
sources:
- name: (R) a unique name for this source entry
from: (R) the provider to use. use `CodeCommit` for CodeCommit
repository: (R) the name of the repository
branch: (R) the branch to build from
connection_arn: (R) the codestar connection ARN to use to pull changes through

Pipeline configuration
The following describes the pipeline's build configuration.
Full schema is documented below, required elements are specified with (R), and defaults are indicated.
stages:
- name: (R) a unique name for this deploy stage, e.g. "Build", "Deploy", etc
enabled: whether or not this stage is enabled (default: true)
actions:
- name: (R) a unique name for this action, e.g. "Build", "DeployToStaging",
"DeleteProduction", etc
provider: the provider to use to perform the action,
allowed values: CodeBuild (default: CodeBuild).
Note: list may be extended in future
buildspec: a path to a CodeBuild buildspec YAML file from your first
repository's root directory
commands: a list of commands to run as part of your buildspec
artifacts: a list of file path artifacts to store after your build
compute_type: the compute type to use (default: config.codebuild.compute_type's value)
image: the docker image to use (default: config.codebuild.image's value)
environment: a hash of "key: value" variables to provide to the build
input_artifacts: a list of other build actions `Name` fields, who's artifacts
to bring in to your build

BuildSpec / Commands / Artifacts
When using CodeBuild, you can either:

not specify buildspec, commands, or artifacts fields, which will default to using buildspec.yml in the primary source's root directory for build instructions,
provide a buildspec field, to specify a file in your repository that contains build instructions,
provide commands (and artifacts if needed) for pipegen to generate a buildspec for you inline.

The template reference for a CodeBuild buildspec can be found here: https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html. If you specify commands, pipegen uses buildspec v0.2 to generate a buildspec inline.
Environment Variables
By default, pipegen provides two environment variables to your build project:
AWS_DEFAULT_REGION and AWS_REGION, both are set to the AWS region that you deploy to (CloudFormation equivalent to AWS::Region). Either of these can be overridden by specifying those keys with alternative values.
You can provide additional environment variables as needed using the following syntax:
environment:
KEY: VALUE
MY_VARIABLE: my value

Input Artifacts
Every CodeBuild project that pipegen configures will have all sources configured added as inputs, meaning that each build will have access to each repository.
If you are building using a "build then deploy" pattern, you may want to pass build artifacts through to deployment stages. You can achieve this with a syntax similar to the following:
stages:
- name: Build
actions:
- name: MyBuildStep
- name: Deploy
actions:
- name: Deploy
commmands:
- bin/deploy
input_artifacts:
- MyBuildStep # << Note that this matches the action name above

Variables and Imports
pipegen supports two special syntaxes for most configuration entries.
Variables
Any --var key=name passed to the CLI can be used inside your configuration file, to vary behaviour at runtime using a syntax like {{vars.KEY}}. This is commonly used to vary things like the branch used. For example:
config.yml:
sources:
- name: Source
from: CodeCommit
repository: my-repo
branch: {{vars.BranchName}}

pipegen cli:
pipegen deploy --var BranchName=main ...

Further, as configuration for pipegen is evaluated using jinja2, you can use variables to determine other values. Commonly this can be used to enable/disable stages based on branch (e.g. only running a production deploy for the main branch).
stages:
- name: Production
enabled: {{vars.BranchName == "main"}}

Imports
pipegen supports importing values from CloudFormation exports using syntax like import:ImportName.
Example:
config:
s3_bucket: import:S3BucketName
kms_key_arn: import:KmsKeyArn

Example configuration
Building from CodeCommit
---
config:
s3_bucket: my-s3-bucket
kms_key_arn: arn:aws:kms:REGION:ACCOUNT:key/KEY-ID

sources:
- name: Source
from: CodeCommit
repository: my-repo
branch: {{vars.BranchName}}

stages:
- name: Build
actions:
- name: Build
provider: CodeBuild
buildspec: buildspecs/build.yml
- name: DevDeploy
actions:
- name: Deploy
provider: CodeBuild
buildspec: buildspecs/deploy.yml
environment:
TARGET_ENVIRONMENT: dev
input_artifacts:
- Build
- name: ProdDeploy
enabled: {{vars.BranchName == "main"}}
actions:
- name: ProdDeploy
provider: CodeBuild
buildspec: buildspecs/deploy.yml
environment:
TARGET_ENVIRONMENT: prod
input_artifacts:
- Build

Building from CodeStar/Github.com
---
config:
s3_bucket: import:S3BucketName
kms_key_arn: import:KMSKeyArn
codebuild:
log_group:
retention: 7 # (days)

sources:
- name: codebuild-test
from: CodeStarConnection
repository: samjarrett/codebuild-test
branch: main
connection_arn: arn:aws:codestar-connections:REGION:ACCOUNT:connection/CONNECTION-ID

stages:
- name: Build
actions:
- name: Build
provider: CodeBuild
commands:
- make build
artifacts:
- build/*
- name: Deploy
actions:
- name: Deploy
provider: CodeBuild
commands:
- make deploy
input_artifacts:
- Build