GitLocker: The Coding Marketplace

Description:

artifactorycleanup 1.0.11

Artifactory cleanup
artifactory-cleanup is an extended and flexible cleanup tool for JFrog Artifactory.
The tool has simple YAML-defined cleanup configuration and can be extended with your own rules on Python.
Everything must be as a code, even cleanup policies!
Tables of Contents

Artifactory cleanup
Tables of Contents
Installation
Usage

Notes
Commands

Rules

Common
Delete
Keep
Docker
Filters
Create your own rule

How to

How to connect self-signed certificates for docker?
How to clean up Conan repository?
How to keep latest N docker images?

Release

Installation
As simple as one command!
# docker
docker pull devopshq/artifactory-cleanup
docker run --rm devopshq/artifactory-cleanup artifactory-cleanup --help

# python (later we call it 'cli')
python3 -mpip install artifactory-cleanup
artifactory-cleanup --help

Usage
Suppose you want to remove all artifacts older than N days from reponame repository.
You should take the following steps:

Install artifactory-cleanup (see above)
Create a configuration file artifactory-cleanup.yaml. variables.

# artifactory-cleanup.yaml
artifactory-cleanup:
server: https://repo.example.com/artifactory
# $VAR is auto populated from environment variables
user: $ARTIFACTORY_USERNAME
password: $ARTIFACTORY_PASSWORD

policies:
- name: Remove all files from repo-name-here older than 7 days
rules:
- rule: Repo
name: "reponame"
- rule: DeleteOlderThan
days: 7

Run the command TO SHOW (not remove) artifacts that will be deleted. By default artifactory-cleanup uses "dry
mode".

# Set the credentials with delete permissions
export ARTIFACTORY_USERNAME=usernamehere
export ARTIFACTORY_PASSWORD=password

# docker
docker run --rm -v "$(pwd)":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup

# cli
artifactory-cleanup

Verify that right artifacts will be removed and add --destroy flag TO REMOVE artifacts:

# docker
docker run --rm -v "$(pwd)":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup --destroy

# cli
artifactory-cleanup --destroy

Looking for more examples? Check examples folder!
Notes

Always specify version of artifactory-cleanup when using it in the production. 1.0.0 is just an example, find the latest version in pypi: https://pypi.org/project/artifactory-cleanup/

# docker
docker pull devopshq/artifactory-cleanup:1.0.0
docker run --rm devopshq/artifactory-cleanup:1.0.0 artifactory-cleanup --version

# python (later we call it 'cli')
python3 -mpip install artifactory-cleanup==1.0.0
artifactory-cleanup --help

Use CI servers or cron-like utilities to run artifactory-cleanup every day (or every hour). TeamCity and GitHub have
built-in support and show additional logs format
Do not save credentials in the configuration file, use environment variables.
Use --ignore-not-found flag to ignore errors when the repository is not found. It's useful when you have a
configuration for multiple repositories and some of them are not found.
Use --worker-count=<WORKER_NUM> to increase the number of workers. By default, it's 1. It's useful when you have a lot of
artifacts and you want to speed up the process.

Commands
# Debug - "dry run" mode by default
# debug run - only print artifacts. it does not delete any artifacts
artifactory-cleanup

# Debug run only for policytestname.
artifactory-cleanup --policy-name policytestname

# REMOVE
# For remove artifacts use --destroy
artifactory-cleanup --destroy

# For remove artifacts use environment variable
export ARTIFACTORY_CLEANUP_DESTROY=True
artifactory-cleanup

# Specify config filename
artifactory-cleanup --config artifactory-cleanup.yaml

# Specify config filename using environment variable
export ARTIFACTORY_CLEANUP_CONFIG_FILE=artifactory-cleanup.yaml
artifactory-cleanup --config artifactory-cleanup.yaml

# Look in the future - shows what the tool WILL remove after 10 days
artifactory-cleanup --days-in-future=10

# Not satisfied with built-in rules? Write your own rules in python and connect them!
artifactory-cleanup --load-rules=myrule.py
docker run -v "$(pwd)":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py

# Save the table summary in a file
artifactory-cleanup --output=myfile.txt

# Save the summary in a json file
artifactory-cleanup --output=myfile.txt --output-format=json

# Save the summary in a json file and append the list of all removed artifacts
artifactory-cleanup --output=myfile.json --output-format json --output-artifacts

Rules
Common

Repo - Apply the rule to one repository. If no name is specified, it is taken from the rule name (in CleanupPolicy
definition)

- rule: Repo
name: reponame

# OR - if you have a single policy for the repo - you can name the policy as reponame
# Both configurations are equal
policies:
- name: reponame
rules:
- rule: Repo

RepoList - Apply the policy to list of repositories.

- rule: RepoList
repos:
- repo1
- repo2
- repo3

RepoByMask - Apply rule to repositories matching by mask

- rule: RepoByMask
mask: "*.banned"

PropertyEq- Delete repository artifacts only with a specific property value (property_key is the name of the
parameter, property_value is the value)

- rule: PropertyEq
property_key: key-name
property_value: 1

PropertyNeq- Delete repository artifacts only if the value != specified. If there is no value, delete it anyway.
Allows you to specify the deletion flag do_not_delete = 1

- rule: PropertyNeq
property_key: key-name
property_value: 1

Delete

DeleteOlderThan - deletes artifacts that are older than N days

- rule: DeleteOlderThan
days: 1

DeleteWithoutDownloads - deletes artifacts that have never been downloaded (DownloadCount=0). Better to use
with DeleteOlderThan rule

- rule: DeleteWithoutDownloads

DeleteOlderThanNDaysWithoutDownloads - deletes artifacts that are older than N days and have not been
downloaded

- rule: DeleteOlderThanNDaysWithoutDownloads
days: 1

DeleteNotUsedSince - delete artifacts that were downloaded, but for a long time. N days passed. Or not
downloaded at all from the moment of creation and it's been N days

- rule: DeleteNotUsedSince
days: 1

DeleteEmptyFolders - Clean up empty folders in given repository list

- rule: DeleteEmptyFolders

DeleteByRegexpName - delete artifacts whose name matches the specified regexp

- rule: DeleteByRegexpName
regex_pattern: "\d"

Keep

KeepLatestNFiles - Leaves the last (by creation time) files in the amount of N pieces. WITHOUT accounting
subfolders

- rule: KeepLatestNFiles
count: 1

KeepLatestNFilesInFolder - Leaves the last (by creation time) files in the number of N pieces in each
folder

- rule: KeepLatestNFilesInFolder
count: 1

KeepLatestVersionNFilesInFolder - Leaves the latest N (by version) files in each
folder. The definition of the version is using regexp. By default it parses semver using the regex - ([\d]+\.[\d]+\.[\d]+)")

- rule: KeepLatestVersionNFilesInFolder
count: 1
custom_regexp: "[^\\d][\\._]((\\d+\\.)+\\d+)"

KeepLatestNupkgNVersions - Leaves N nupkg (adds *.nupkg filter) in release feature builds

- rule: KeepLatestNupkgNVersions
count: 1

Docker

DeleteDockerImagesOlderThan - Delete docker images that are older than N days

- rule: DeleteDockerImagesOlderThan
days: 1

DeleteDockerImagesOlderThanNDaysWithoutDownloads - Deletes docker images that are older than N days and have
not been downloaded

- rule: DeleteDockerImagesOlderThanNDaysWithoutDownloads
days: 1

DeleteDockerImagesNotUsed - Removes Docker image not downloaded since N days

- rule: DeleteDockerImagesNotUsed
days: 1

IncludeDockerImages - Apply to docker images with the specified names and tags

- rule: IncludeDockerImages
masks: "*singlemask*"
- rule: IncludeDockerImages
masks:
- "*production*"
- "*release*"

ExcludeDockerImages - Exclude Docker images by name and tags.

- rule: ExcludeDockerImages
masks:
- "*production*"
- "*release*"

KeepLatestNVersionImagesByProperty(count=N, custom_regexp='some-regexp', number_of_digits_in_version=X) - Leaves N
Docker images with the same major. (^\d+\.\d+\.\d+$) is the default regexp how to determine version which matches semver 1.1.1. If you
need to add minor then set number_of_digits_in_version to 2 or if patch then set to 3 (by default we match major, which 1). Semver tags
prefixed with v are supported by updating the regexp to include (an optional) v in the expression (e.g., (^v?\d+\.\d+\.\d+$)).

- rule: KeepLatestNVersionImagesByProperty
count: 1
custom_regexp: "[^\\d][\\._]((\\d+\\.)+\\d+)"

DeleteDockerImageIfNotContainedInProperties(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None)
- Remove Docker image, if it is not found in the properties of the artifact repository.

DeleteDockerImageIfNotContainedInPropertiesValue(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None)
- Remove Docker image, if it is not found in the properties of the artifact repository.

Filters

IncludePath - Apply to artifacts by path / mask.

- rule: IncludePath
masks: "*production*"
- rule: IncludePath
masks:
- "*production*"
- "*develop*"

IncludeFilename - Apply to artifacts by name/mask

- rule: IncludeFilename
masks:
- "*production*"
- "*develop*"

ExcludePath - Exclude artifacts by path/mask

- rule: ExcludePath
masks:
- "*production*"
- "*develop*"

ExcludeFilename - Exclude artifacts by name/mask

- rule: ExcludeFilename
masks:
- "*.tag.gz"
- "*.zip"

Create your own rule
If you want to create your own rule, you can do it!
The basic flow how the tool calls Rules:

Rule.check(*args, **kwargs) - verify that the Rule configured right. Call other services to get more information.
Rule.aql_add_filter(filters) - add Artifactory Query Language expressions
Rule.aql_add_text(aql) - add text to the result aql query
artifactory-cleanup calls Artifactory with AQL and pass the result to the next step
Rule.filter(artifacts) - filter out artifacts. The method returns artifacts that will be removed!.

To keep artifacts use artifacts.keep(artifact) method

Create myrule.py file at the same folder as artifactory-cleanup.yaml:
# myrule.py
from typing import List

from artifactory_cleanup import register
from artifactory_cleanup.rules import Rule, ArtifactsList

class MySimpleRule(Rule):
"""
This doc string is used as rule title

For more methods look at Rule source code
"""

def __init__(self, my_param: str, value: int):
self.my_param = my_param
self.value = value

def aql_add_filter(self, filters: List) -> List:
print(f"Today is {self.today}")
print(self.my_param)
print(self.value)
return filters

def filter(self, artifacts: ArtifactsList) -> ArtifactsList:
"""I'm here just to print the list"""
print(self.my_param)
print(self.value)
# You can make requests to artifactory by using self.session:
# url = f"/api/storage/{self.repo}"
# r = self.session.get(url)
# r.raise_for_status()
return artifacts

# Register your rule in the system
register(MySimpleRule)

Use rule: MySimpleRule in configuration:
# artifactory-cleanup.yaml
- rule: MySimpleRule
my_param: "Hello, world!"
value: 42

Specify --load-rules to the command:
# docker
docker run -v "$(pwd)":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py

# cli
artifactory-cleanup --load-rules=myrule.py

How to
How to connect self-signed certificates for docker?
In case you have set up your Artifactory self-signed certificates, place all certificates of the chain of trust into
the certificates folder and add additional argument to the command:
docker run -v "$(pwd)":/app -v "$(pwd)/certificates":/mnt/self-signed-certs/ devopshq/artifactory-cleanup artifactory-cleanup

How to clean up Conan repository?
We can handle conan's metadata by creating two policies:

First one removes files but keep all metadata.
Second one look at folders and if it contains only medata files - removes it (because there's no associated with
metadata files)

The idea came from https://github.com/devopshq/artifactory-cleanup/issues/47
# artifactory-cleanup.yaml
artifactory-cleanup:
server: https://repo.example.com/artifactory
user: $ARTIFACTORY_USERNAME
password: $ARTIFACTORY_PASSWORD

policies:
- name: Conan - delete files older than 60 days
rules:
- rule: Repo
name: "conan-testing"
- rule: DeleteNotUsedSince
days: 60
- rule: ExcludeFilename
masks:
- ".timestamp"
- "index.json"
- name: Conan - delete empty folders (to fix the index)
rules:
- rule: Repo
name: "conan-testing"
- rule: DeleteEmptyFolders
- rule: ExcludeFilename
masks:
- ".timestamp"
- "index.json"

How to keep latest N docker images?
We can combine docker rules with usual "files" rules!
The idea came from https://github.com/devopshq/artifactory-cleanup/issues/61
# artifactory-cleanup.yaml
artifactory-cleanup:
server: https://repo.example.com/artifactory
user: $ARTIFACTORY_USERNAME
password: $ARTIFACTORY_PASSWORD

policies:
- name: Remove docker images, but keep last 3
rules:
# Select repo
- rule: Repo
name: docker-demo
# Delete docker images older than 30 days
- rule: DeleteDockerImagesOlderThan
days: 30
# Keep these tags for all images
- rule: ExcludeDockerImages
masks:
- "*:latest"
- "*:release*"
# Exclude these docker tags
- rule: ExcludePath
masks: "*base-tools*"
# Keep 3 docker tags for all images
- rule: KeepLatestNFilesInFolder
count: 3

Release
In order to provide a new release of artifactory-cleanup, there are two steps involved.

Bump the version in the setup.py
Bump the version in the init.py
Create a Git release tag (in format 1.0.1) by creating a release on GitHub