marchitect 0.8

Description:

marchitect 0.8

Marchitect
A tool for uploading files to and running commands on remote hosts.
Install
$ pip3 install marchitect

Example
Let's write a couple of files to your machine assuming that it could easily be
made to be a remote machine.
from marchitect.site_plan import SitePlan, Step
from marchitect.whiteprint import Whiteprint

class HelloWorldWhiteprint(Whiteprint):

name = 'hello_world'

def _execute(self, mode: str) -> None:
if mode == 'install':
# Write file by running remote shell commands.
self.exec('echo "hello, world." > /tmp/helloworld1')
# Write file by uploading
self.scp_up_from_bytes(
b'hello, world.', '/tmp/helloworld2')

class MyMachine(SitePlan):
plan = [
Step(HelloWorldWhiteprint)
]

if __name__ == '__main__':
# SSH into your own machine, prompting you for your password.
import getpass
import os
user = os.getlogin()
password = getpass.getpass('%s@localhost password: ' % user)
sp = MyMachine.from_password('localhost', 22, user, password, {}, [])
# If you want to auth by private key, use the below:
# (Note: The password prompt will be for your private key, empty for none.)
#sp = MyMachine.from_private_key(
# 'localhost', 22, user, '/home/%s/.ssh/id_rsa' % user, password, {}, [])
sp.install() # Sets the mode of _execute() to install.

This example requires that you can SSH into your machine via password. To use
your SSH key instead, uncomment the lines above. After execution, you should
have /tmp/helloworld1 and /tmp/helloworld2 on your machine.
Hopefully it's clear that whiteprints let you run commands and upload files to a
target machine. A whiteprint should contain all the operations for a common
purpose. A site plan contains all the whiteprints that should be run on a
single machine class.
Steps for deploying your code repository to a machine would make for a good
whiteprint. A site plan for a machine that runs your web servers might use that
whiteprint and others.
Goals

Easy to get started.
Templating of configuration files.
Mix of imperative and declarative styles.
Arbitrary execution modes (install, update, clean, start, stop, ...).
Interface for validating machine state.
Be lightweight because most complex configurations are happening in
containers anyway.

Non-goals

Making whiteprints and site plans share-able with other people and companies.
Non-Linux deployment targets.

Concepts
Whiteprint
To create a whiteprint, extend Whiteprint and define a name class variable
and an _execute() method; optionally define a validate() method. name
should be a reasonable name for the whiteprint. In the example above, the
HelloWorldWhiteprint class's name is simply hello_world. name is
important for file resolution which is discussed below.
_execute() is where all the magic happens. The method takes a string called
mode. Out of convention, your whiteprints should handle the following modes:

install (installing software)
update (updating software)
clean (removing software, if needed, but generally impractical)
start (starting services)
stop (stopping services).

Despite this convention, mode can be anything as you'll be choosing the modes
to execute your site plans with.
Within _execute(), you're given all the freedom to shoot yourself in the
foot. Use self.exec() to run any command on the target machine.
exec() returns an ExecOutput object with variables exit_status (int),
stdout (bytes), and stderr (bytes). You can use these outputs to control
flow. If the exit status is non-zero, a RemoteExecError is raised. To
suppress the exception, set error_ok=True.
_execute() has access to self.cfg which are the config variables for the
whiteprint. See the Templates & Config Vars section below.
Use the variety of functions to copy files to and from the host:

scp_up() - Upload a file from the local host to the target.
sp_up_from_bytes() - Create a file on the target host from the bytes arg.
scp_down() - Download a file from the target to the local host.
scp_down_to_bytes() - Download a file from the target and return it.

Templates & Config Vars
You can upload files that are jinja2 templates. The
templates will be filled by the config variables passed to the whiteprint.
Config variables can be set in a few ways, which we'll explore.
Here's a sample test.toml file that uses the jinja2 notation to specify a
name variable with a default of John Doe:
name = "{{ name|default('John Doe') }}"

A whiteprint can populate a template for upload as follows:
from marchitect.whiteprint import Whiteprint

class WhiteprintExample(Whiteprint):

default_cfg = {'name': 'Alice'}

def _execute(self, mode: str) -> None:
if mode == 'install':
self.scp_up_template('/path/to/test.toml', '~/test.toml')

A whiteprint can also upload a populated template that's stored in a string
rather than a file:
from marchitect.whiteprint import Whiteprint

class WhiteprintExample(Whiteprint):

default_cfg = {'name': 'Alice'}

def _execute(self, mode: str) -> None:
if mode == 'install':
self.scp_up_template_from_str(
'name = "{{ name }}"', '~/test.toml')

A config var can be overriden in scp_up_template_from_str:
from marchitect.whiteprint import Whiteprint

class WhiteprintExample(Whiteprint):

default_cfg = {'name': 'Alice'}

def _execute(self, mode: str) -> None:
if mode == 'install':
# 'Bob' overrides 'Alice'
self.scp_up_template_from_str(
'name = "{{ name }}"', '~/test.toml',
cfg_override={'name': 'Bob'})

Config vars can also be set by the SitePlan in the plan or during
instantiation.
from marchitect.site_plan import Step, SitePlan

class MyMachine(SitePlan):
plan = [
Step(WhiteprintExample, {'name': 'Eve'})
]

if __name__ == '__main__':
MyMachine.from_password(..., cfg={WhiteprintExample: {'name': 'Foo'}})

In the above, Foo takes precedence over Eve which takes precedence over any
values for name defined in the whiteprint.
Config Override by Alias
Finally, a Step can be given an alias as another identifier for specifying
config vars. This is useful when a whiteprint is used multiple times in a site
plan.
from marchitect.site_plan import Step, SitePlan

class MyMachine(SitePlan):
plan = [
Step(WhiteprintExample, alias="ex1"),
Step(WhiteprintExample, alias="ex2"),
]

if __name__ == '__main__':
MyMachine.from_password(..., cfg={'ex1': 'Eve', 'ex2': 'Foo'})

In the above, the first WhiteprintExample uploads Eve and the second
replaces it with Foo.
Auto-Derived Configs
Auto-derived config variables are always available without specification.
These are stored in self.cfg['_target']:

user: The login user for the SSH connection.
host: The target host for the SSH connection.
kernel: The kernel version of the target host. Ex: 4.15.0-43-generic
distro: The Linux distribution of the target host. Ex: ubuntu
disto_version: The version of the Linux distribution. Ex: 18.04
hostname: The hostname of the target host.
fqdn: The fully-qualified domain name of the target host.
cpu_count: The number of CPUs on the target host. Ex: 8

Config Var Schema
Because config vars may be used in external template files, it's not readily
observable what vars are used by a whiteprint. To make config vars explicit,
a schema can be set using cfg_schema:
from marchitect.whiteprint import Whiteprint
import schema

class WhiteprintExample(Whiteprint):

cfg_schema = {
'name': str,
schema.Optional('path'): str,
'targets': [str],
}
...

The schema is enforced on execution of the whiteprint.
For more info on expressing schemas (nesting, lists, optionals), see
schema.
File Resolution
Methods that upload local files (scp_up() and scp_up_template()) will
search for the files according to the rsrc_paths argument in the SitePlan
constructor. The search proceeds in order of the rsrc_paths and the name of
the whiteprint is expected to be the name of a subfolder in the rsrc_path.
For example, assume rsrc_paths is [Path('/srv/rsrcs')], the whiteprint
has a name of foobar, and the file c is referenced as a/b/c. The resolver
will look for the existence of /srv/rsrcs/foobar/a/b/c.
If a file path is specified as absolute, say /a/b/c, no rsrc_path will be
prefixed. However, this form is not encouraged for portability across machines
as resources may live in different folders on different machines.
Idempotence
It's important to strive for the idempotence of your whiteprints. In other
words, assume your whiteprint in any mode (install, update, ...) can be
interrupted at any point. Can your whiteprint be re-applied successfully
without any problems?
If so, your whiteprint is idempotent and is therefore resilient to connection
errors and software hiccups. Error handling will be as easy as retrying your
whiteprint a bounded number of times. If not, you'll need to figure out an
error handling strategy. In the extreme case, you can terminate servers that
produce errors and start over with a fresh one, assuming that you're in a cloud
environment.
Prefab
Prefabs are built-in, robust whiteprints you can use in your whiteprints.
These make it easy to add common functionality with the _execute() and
_validate() methods already defined. These are available out-of-the-box:

Apt: Common Linux package manager.
Pip3: Python package manager.
Folder: Makes a folder exists at the specified path.
LineInFile: Ensures the specified line exists in the specified file.
FileFromString: Makes a file at a specified path.
FileFromPath: Makes a file at a specified path.
Symlink: Makes a symlink.
FileExistsValidator: Only validates that a file exists at a specified path.

An example:
from marchitect.prefab import Apt
from marchitect.whiteprint import Prefab, Whiteprint

class HelloWorld2Whiteprint(Whiteprint):

prefabs_head = [
Prefab(Apt, {'packages': ['curl']}),
]

def _execute(self, mode: str) -> None:
if mode == 'install':
self.exec('curl https://www.nytimes.com > /tmp/nytimes')

prefabs_head are applied before your _execute() and _validate() methods,
respectively. Alternatively, prefabs_tail are applied after.
If a prefab depends on a config variable, define a _compute_prefabs_head()
class method:
from typing import Any, Dict, List
from marchitect.prefab import Folder
from marchitect.whiteprint import Prefab, Whiteprint

class ExampleWhiteprint(Whiteprint):

cfg_schema = {
'temp_folder': str,
}

@classmethod
def _compute_prefabs_head(cls, cfg: Dict[str, Any]) -> List[Prefab]:
return [Prefab(Folder, {'path': cfg['temp_folder']})]

The prefabs returned by_compute_prefabs_head() will be applied after those
specified directly in the prefabs_head class variable.
Nested Whiteprints
Whiteprints can use other whiteprints.
from marchitect.whiteprint import Whiteprint

class Example2Whiteprint(Whiteprint):
pass

class ExampleWhiteprint(Whiteprint):
def _execute(self, mode: str) -> None:
if mode == 'install':
self.use_execute(mode, Example2Whiteprint, {})

def _validate(self, mode: str) -> None:
if mode == 'install':
self.use_validate(mode, Example2Whiteprint, {})

Site Plan
Site plans are collections of whiteprints. You likely have distinct roles for
the machines in your infrastructure: web hosts, api hosts, database hosts, ...
Each of these should map to their own site plan which will install the
appropriate whiteprints (postgres for database hosts, uwsgi for web hosts, ...).
Testing
Tests are run against real SSH connections, which unfortunately makes it
difficult to run in a CI environment. Travis CI is not enabled for this reason.
When running tests through py.test or tox, you can specify SSH credentials
either as a user/pass pair or user/private_key. For example:
SSH_USER=username SSH_HOST=localhost SSH_PRIVATE_KEY=~/.ssh/id_rsa SSH_PRIVATE_KEY_PASSWORD=*** py.test
SSH_USER=username SSH_HOST=localhost SSH_PASSWORD=*** py.test -sx

Will likely move to mocking SSH commands, but it will be painful to reliably
mock the interfaces for ssh2-python.
mypy and lint are also supported: tox -e mypy,lint
TODO

Add "common" dependencies to minimize invocations of commands like
apt update to once per site plan.
Write a log of applied site plans and whiteprints to the target host
for easy debugging.
Add documentation for validate() method.
Verify speed wins by using ssh2-python instead of paramiko.
Document SitePlan.one_off_exec().
File prefabs can use md5sum to decide whether to re-create file.