GitLocker: The Coding Marketplace

Description:

impulsarejob 1.0a2

impulsare/job

Overview
A jobs manager, specific to impulsare. It reads, write and delete jobs from an sqlite db.
See tests/static/ for examples of configuration.

Installation / Usage
To install use pip:
$ pip install --upgrade impulsare-job

Configuration
You need to create a configuration file that contains:
job:
db: /tmp/test.db # required

Architecture

Writer
impulsare/job implements a writer to :

Create / Update jobs (save())
Delete jobs (delete())
Add / Remove Hooks (hooks_writer.add_hook() and hooks_writer.del_hook())
Add / Remove Fields (fields_writer.add_field() and fields_writer.del_field())
Add / Remove Rules related to Fields (fields_writer.add_rule() and fields_writer.del_rule())

Reader
And a Reader to :

Get a Job
Get related hooks
Get related fields + their rules

Properties of a job
{
'name': str, # required
'active': bool, # default : True
'description': str,
'priority': int, # default : 1
'input': str, # required
'input_parameters': dict,
'output': str, # required
'output_parameters': dict,
'mode': str # c (create), u (update), cu (create/update), d (delete). Default: c
}

Examples

Create a simple Job (no hooks / rules)
from impulsare_job import Writer

writer = Writer('/etc/impulsare/config.yml')
writer.set_prop('name', 'My Job')
writer.set_prop('input', 'csv')
writer.set_prop('input_parameters', {'delimiter': 'csv'})
writer.set_prop('output', 'sql')
writer.set_prop('output_parameters', {'db': 'test'})
job = writer.save()

Update a Job
from impulsare_job import Writer

# Lets assume the job id = 1
writer = Writer('/etc/impulsare/config.yml', 'My Job')
job = writer.get_job()
print(job.name)
# Output: 'My Job'

# Set the job to Inactive
writer.set_prop('active', False)
writer.save()

Verify if a hook exists, else add it
# .... continuation of code above
if not writer.hooks_writer.hook_exists('test'):
writer.hooks_writer.add_hook(name='upload_file', method='upload_file', when='after_process')
Allowed properties for hooks:
{
'name': str, # required
'method': str, # required
'when': str, # required
'description': str,
'active': bool, # Default : True
'priority': int # Default: 1
}
Other methods:

hooks_writer.get_hooks
hooks_writer.del_hook

There is no method update, to update a hook, delete it then recreate it.

Verify if a field exists, else update it and add a transformation rule
Warning : a field is identified by its output value that must be unique
(we can’t send two values for the same field while we can use the same input field
for various output).
Field
# .... continuation of code above
if writer.fields_writer.field_exists('firstname'):
writer.fields_writer.del_field('firstname')

writer.fields_writer.add_field(input='first_name', output='firstname')
Allowed properties for fields:
{
'input': str, # required
'output': str, # required
}
Other methods:

fields_writer.get_field
fields_writer.get_fields

There is no method update, to update a field, delete it then recreate it.

Add a rule
writer.fields_writer.add_rule(output_field='firstname', name='uppercase', method='uppercase')
Allowed properties for rules:
{
'name': str, # required
'method': str, # required
'description': str,
'active': bool, # Default : True
'params': list,
'blocking': bool, # Default : False
'priority': int # Default: 1
}
Other methods:

fields_writer.del_rule
fields_writer.get_rules
fields_writer.rule_exists

There is no method update, to update a rule, delete it then recreate it.

Retrieve a Job, its hooks and fields
from impulsare_job import Reader

Reader = Reader('/etc/impulsare/config.yml', 'My Job')
job = Reader.get_job()
hooks = Reader.get_hooks()
fields = Reader.get_fields() # Get rules for first field : rules = fields[0].rules

Development & Tests
$ pip install -r requirements.txt
$ pip install -r requirements-dev.txt
$ py.test

TODO

Don’t check if table exists on each model but do it on app installation
To get the DB :
from impulsare_job import models

db = models.get_db('/etc/impulsare/config.yml')
db.create_tables([models.Job, models.Hook, models.Rule])

Refactor writer
To have a class for hooks, and another for rules.