Description:
gitlabjobguard 0.0.4.3
gitlab-job-guard
Guard Gitlab CI pipeline jobs from multiple parallel executions.
$ PRIVATE_TOKEN="$GITLAB_API_TOKEN" gitlab-job-guard
$ my-unguarded-deployment-task --to=production
gitlab-job-guard will block if it detects other pipelines running for
the current project to avoid multiple pipelines from clobbering up a
deployment/environment. This is especially a problem when jobs are distributed
and picked up by multiple gitlab runners.
While gitlab will auto-cancel redundant, pending pipelines for the same branch
by default - this is not the case for multiple pipelines from different
branches targeting a particular deployment/environment. Gitlab has no way to
detect or control these user-defined branch-to-environment mappings and this
means environments can easily be left in an unsafe/broken state. (e.g.
terraform apply or ansible, etc from different pipelines running at the same
time).
gitlab-job-guard uses the Gitlab API to determine if existing pipelines are
scheduled and to backoff-and-retry until it is safe to proceed even if jobs are
distributed across multiple runners. Conflicts are detected by user-defined
matches on pipeline ref names (branch, tag, etc) and/or pipeline status.
Usage
The simplest use-case would likely be placing gitlab-job-guard in a
before_script section in your gitlab-ci.yml to protect all jobs (though this
can slow things down and be unfriendly to your local gitlab-ops).
$GITLAB_API_TOKEN is a Personal Access
Token
with at minimum api scope.
before_script:
- PRIVATE_TOKEN="$GITLAB_API_TOKEN" gitlab-job-guard
Though often, most pipelines only need to guard the critical jobs that share
common state/data (i.e. a provisioning/deployment job, an artifact build/release
job, etc).
deploy-production:
stage: deploy
script:
- PRIVATE_TOKEN="$GITLAB_API_TOKEN" gitlab-job-guard
- my-unguarded-deployment-task --to=production
e.g To guard something critical like a terraform job running for a tag.
provision-infrastructure:
stage: provision
script:
- export PRIVATE_TOKEN="$GITLAB_API_TOKEN"
- gitlab-job-guard --guard-ref-regex='^v[0-9\.]+' # Regex matches tags
- terraform plan ...
- terraform apply ...
only:
- tags
Other usages
--guard-ref-regex takes a regular expression to detect conflicts on other
pipelines whose ref name (branch, tag) is a (partial) match. This allows for
guarding of jobs in the complex use-cases.
To hold jobs for a collisions on pattern matches on the branch/tag names.
gitlab-job-guard -c=^master$ # Match branch names matching 'master' exactly
gitlab-job-guard -c=^(master|dev(elop)?)$ # Match any of the mainline branches
gitlab-job-guard -c=^(feature|release|hotfix)/ # Match any gitflow transient branch prefixes
gitlab-job-guard -c=^[0-9]\- # Match branch names beginning with a number
# and dash ignoring all other text.
# e.g. a gitlab branch made from an issue
gitlab-job-guard -c=^v?[\d.]+$ # Match (semver) tags like v1.0.9, 2.0
gitlab-job-guard -c=^environment/ # Match any environment deployments?
gitlab-job-guard -c=^environment/dc1.+ # Match environment deployments to DC1?
gitlab-job-guard -c="$CI_BUILD_REF_NAME" # Match current branch name (partially).
# i.e. 'master' matches 'feature/master-document'
gitlab-job-guard -c="^$CI_BUILD_REF_NAME$" # Match current branch name (exactly).
# i.e. 'master' does not match 'master-deployment'
gitlab-job-guard -c='.+' -s='running|pending' # Match any pipeline in running or pending state
To hold a job for a collision on part of the ref name (e.g. on branch prefix
such as feature/ or hotfix/ or release/, etc a la gitflow).
# Assuming current pipeline is for the 'feature/foo' branch and that
# CI_BUILD_REF_NAME=feature/foo
# Extract the branch prefix
CI_BUILD_REF_PREFIX=$(echo "$CI_BUILD_REF_NAME" | sed -r 's@(.+/)(.+)@\1@')
# CI_BUILD_REF_PREFIX now contains 'feature/'
# This will now block if there are any other pipelines running or pending
# for feature/ branches.
gitlab-job-guard -c="^$CI_BUILD_REF_PREFIX" -s='running|pending'
Other arguments
Typically gitlab-job-guard reads in the following environment variables
to compose its arguments. NOTE: environment variables take precedence
over arguments passed at the CLI and so only the user-specified
arguments/envvars need to be set. Setting these as protected environment
variables avoids
leaking sensitive information and needing to duplicate them for every call.
envvar | argument | source
------------------- --------------------- ----------------
GUARD_REF_REGEX | --guard-ref-regex | user specified
GUARD_STATUS_REGEX | --guard-status-regex | user specified
PRIVATE_TOKEN | --private-token | user specified
GUARD_TIMEOUT | --guard-timeout | user specified
CI_PROJECT_ID | --ci-project-id | runner environment
CI_BUILD_REF_NAME | --ci-build-ref-name | runner environment
CI_API_V4_URL | --ci-api-v4-url | runner environment
CI_PIPELINE_ID | --ci-pipeline-id | runner environment
See gitlab-job-guard -h for the full argument set and current/default values.
Behaviour
gitlab-job-guard will backoff-and-retry until it is safe to proceed or a
timeout is reached. An exit code of 0 indicates no other pipelines conflict
and that the job can continue. On reaching the timeout (1 hour by default to
align with the default timeout for gitlab jobs), it will return an error to fail
the job. The timeout in seconds can be set via -w/--guard-timeout.
If a fail-fast approach is needed, the -x/--no-wait flag will cause
gitlab-job-guard to exit immediately on conflicts. (Though, this means the
pipeline run will have to be restarted some other way or abandoned).
TODO
For long-running pipelines, this solution can have subtle consequences with
growing queues and increased contention and unpredictability as to which
pipeline is the first-past-the-post. An older pipeline taking precedence over
newer commits if often not desired and newer pipelines always winning is
probably desired.
Handle existing conflicting pipelines - cancel them or give-way.
Narrow down conflicts to jobs (CI_JOB_NAME) or stages (CI_JOB_STAGE)
so that other parts of the pipelines that do not share state are allowed to
run freely.
License
For personal and professional use. You cannot resell or redistribute these repositories in their original state.
Customer Reviews
There are no reviews.
