GitLocker: The Coding Marketplace

Description:

gobletworkflows 0.1.4

Goblet Workflows (Beta)

Goblet Workflows is a wrapper around GCP Workflows, which is a fully-managed orchestration platform that executes services in an order that you define: a workflow. These workflows can combine services including custom services hosted on Cloud Run or Cloud Functions, Google Cloud services such as Cloud Vision AI and BigQuery, and any HTTP-based API.
While GCP Workflows is powerful, it requires learning a complex yaml DSL, which makes it hard to manage and scale, and has a steap learning curve to get started.
Goblet Workflows uses similar syntax that you can expect from airflow and prefect, to make it easy to write, deploy, and manager your workflows, while leveraging the severless nature of GCP workflows.
See below for a simple workflow example.
from goblet_workflows.workflow import Workflow
from goblet_workflows.steps import AssignStep, BQStep

w = Workflow("bigquery-step-example", params=["column"])

upack_args = AssignStep(
w,
"upack_args",
column="${column}"
)

bq_step = BQStep(w, "bq_step", "select {column} from DATASET limit 1")

bq_step_2 = BQStep(w, "bq_step_2", "select colomn2 from DATASET2 where column1 = {bq_step[0]} ")

upack_args > bq_step > bq_step_2

Not to deploy you simply have to run the command goblet_workflows deploy, which will yield the workflow below.

Getting started
To install run:
pip install goblet_workflows
To deploy a workflow create a main.py file where you define your workflow and steps.
Then run
goblet_workflows deploy
To view the yaml run
goblet_workflows print
To execute a workflow run
goblet_workflows execute
To execute a workflow run with an arg
goblet_workflows execute --arg '{"project_id":"PROJECT","var1":"xx"}'
Examples
Goblet Workflow Examples
Features

Generic Steps
Branches
Deployment of Workflows and Schedules
GCP Connectors
Execute Workflows

GCP Workflows
Overview
Syntax Cheat Sheet
Issues
Please file any issues, bugs or feature requests as an issue on our GitHub page.
Roadmap
☑ Http Steps
☑ GCP Generic Connectors
☑ BQ Connector
☑ Dataform Connector
☑ Deploy
☑ Deploy Arbitrary File
☑ Execute
☐ Delete
☑ Schedule
☐ Conditions
☐ Subworkflows
☐ Parameter typing
Want to Contribute
If you would like to contribute to the library (e.g. by improving the documentation, solving a bug or adding a cool new feature) submit a pull request.