GitLocker: The Coding Marketplace

Description:

pipengcs 0.0.1

pipen-gcs
A plugin for pipen to handle files in Google Cloud Storage
Installation
pip install -U pipen-gcs

# uninstall to disable
pip uninstall pipen-gcs

Usage
from pipen import Proc, Pipen

class MyProc(Proc):
input = "infile:file"
input_data = ["gs://bucket/path/to/file"]
output = "outfile:file:gs://bucket/path/to/output"
script = "cat {{infile}} > {{outfile}}"

class MyPipen(Pipen):
starts = MyProc
# input files/directories will be downloaded to /tmp
# output files/directories will be generated in /tmp and then uploaded
# to the cloud storage
plugin_opts = {"gcs_localize": "/tmp"}

if __name__ == "__main__":
MyPipen().run()

You can also disable localization, then you will have to handle the
cloud storage files yourself.
from pipen import Proc, Pipen

class MyProc(Proc):
input = "infile:file"
input_data = ["gs://bucket/path/to/file"]
output = "outfile:file:gs://bucket/path/to/output"
script = "gsutil cp {{infile}} {{outfile}}"

class MyPipen(Pipen):
starts = MyProc
plugin_opts = {"gcs_localize": False}

if __name__ == "__main__":
MyPipen().run()

Configuration

gcs_localize: The directory to localize the cloud storage files. If
set to False, the files will not be localized. Default is False.
gcs_credentials: The path to the Google Cloud Service Account
credentials file.