GitLocker: The Coding Marketplace

Description:

aptrepoman 1.0.7

Repoman: a scalable apt server using Amazon SimpleDB and S3
Introduction
Back in 2011, a scrappy young startup called "Weatherbill" found itself with a
conundrum: we were using Debian's dpkg and apt tools to package and serve
binary software within our own infrastructure, but we were also starting to use
Amazon's Elastic Mapreduce to create ephemeral Hadoop clusters for geospatial
and weather data processing, and it turned out that spinning up a 2,000-node
Hadoop cluster and downloading several dozen gigabytes of packages from our
standalone apt server onto each Hadoop compute node at once was an excellent
way to make the apt server vanish in a puff of smoke.
So with the optimism that comes from youth and too much caffiene, we wrote our
own apt server, based on Amazon's Simple Storage Service (S3) and SimpleDB
offerings. We called it "Repoman", because the life of a repo man, much like
that of an engineer at a small startup, is always intense. And since then,
Repoman has successfully served millions of packages to various servers and
services here at The Climate Corporation (which Weatherbill became) and we're
happy to finally get a chance to share it with you.
Requirements
In order to operate an apt repository with Repoman, you will need at
a minimum:

A working Python 2.7 or 3.5 installation
An Amazon Web Services account
User credentials in that AWS account, either in your shell environment,
a ~/.aws/Credentials file, or via an EC2 Instance Profile or ECS Task
Role. In general if the AWS CLI runs
successfully in your environment, Repoman should run.
An AWS SimpleDB Domain
which Repoman will use to store metadata about your packages.
An AWS S3 Bucket
which Repoman will use to store your actual .deb package files and
also the generated metadata files that form an Apt repository.
Your user credentials must have sufficient permissions (via Amazon IAM)
to manipulate both the SimpleDB domain and the S3 bucket: you will need at a
minimum to be able to create and delete keys in both. A sample IAM security
policy is provided in the docs folder

Some basic familiarity with the concepts and nomenclature of running an apt
server is assumed: if you don't know what distributions, components and
architectures are, you may want to review the apt
documentation
Optional

Repoman can function with your own user credentials, or it can assume an IAM
role before querying AWS APIs; see the installation section
for details on how to use roles.
Repoman can sign your apt repository's metadata files
in order to provide strong assurances that the packages in the repo
come from their claimed source. You will need a working GPG keyring,
and key management is not provided by Repoman.

Commands, flags and help
The repoman-cli cli tool offers the following commands:

setup -- initial setup of the repository
checkup -- check that SimpleDB and S3 are configured correctly
add -- add packages to the repository
rm -- delete packages from the repository
cp -- copy packages from one distribution or component in the
repository to another distribution or complnent
query -- list packages in the repository based on filters
publish -- publish the current SimpleDB repository to state to S3
backup -- backup the current SimpleDB state to a JSON file
restore -- restore SimpleDB state from a JSON file
repo -- repository management sub-commands:

repo add-distribution -- add a distribution for the repo to serve
repo rm-distribution -- remove a distribution for the repo to serve
repo add-component -- add a component for the repo to serve
repo rm-component -- remove a component for the repo to serve
repo add-architecture -- add a architecture for the repo to serve
repo rm-architecture -- remove a architecture for the repo to serve
repo add-topic -- add an SNS topic to log notifications to
repo rm-topic -- remove any configured SNS topic
repo show-config -- show the current repository configuration
repo add-origin -- set an Origin string for the published repository
repo add-label -- set a Label string for the published repository

The repoman-cli utility itself and all of its commands and sub-commands will
take a -h or --help flag to show help text and all locally relevant flags.
Most commands that mutate the repository will prompt for confirmation;
this step can be bypassed by passing the -y or --confirm flag.
Some particularly dangerous commands (e.g. deleting an entire distribution's
worth of packages) will prompt for an extra confirmation step; this too
can be bypassed by passing in the --i-fear-no-evil flag but this is a
strictly at-your-own-risk proposition.
To automatically publish the repository to s3 after adding, copying or
removing packages, pass the --publish flag.
Further Documentation

Basic operations

Theory of Operation
Installation, Configuration and Setup
Adding binary and source packages to Repoman
Querying the repository
Publishing and signing the repository
Copying and promoting packages within the repository
Removing and purging pacakges within the repository

Advanced topics

Adding and removing sections (distributions, components, architectures) from the repository
Subscribing apt clients to the repository
Serving public repositories directly from S3
Backups and restores
Logging and notifications
Recovering deleted packages