GitLocker: The Coding Marketplace

Description:

powerfulpipes 1.1.3

In a nutshell Powerful Pipes is a library for working with UNIX Pipes.

Install
> pip install powerful-pipes

Quick Start
Create a CLI tool that reads from the JSON from stdin and dumps to the stdout, after processing input data:
# File: pipe-example.py
from powerful_pipes import read_json_from_stdin, eprint, write_json_to_stdout

for error, input_json in read_json_from_stdin():

if error:
# Here you can manager the error. Most common error is that the
# input is not a valid json.
# eprint(...) function dumps the error message to the stderr
eprint(message="Error processing input from stdin")
continue

try:
input_json["myData"] = "data 1"

except Exception as e:
report_exception(input_json, e)

finally:
# THIS STEP IS CRITICAL. If you don't put again in the stdout the
# input data, following tools in the pipe chain wouldn't receive
# the data
write_json_to_stdout(input_json)
Using in CLI:
> echo '{}' | python pipe-example.py | jq
{
"myData": "data 1"
}

Advanced Usage
If you need to pass heavy data between tools in the pipe chain, you should use the write_to_stdout_by_file_ref and read_from_stdin_by_file_ref.
It stores the data in a temporary file and passes the file reference to the next tool in the pipe chain.
# File: pass_by_reference.py
from powerful_pipes import read_from_stdin_by_file_ref, eprint, write_to_stdout_by_file_ref, report_exception

for error, input_json in read_from_stdin_by_file_ref():

if error:
# Here you can manager the error. Most common error is that the
# input is not a valid json.
# eprint(...) function dumps the error message to the stderr
eprint(message="Error processing input from stdin")
continue

try:
input_json["myData"] = "data 1"

except Exception as e:
report_exception(input_json, e)

finally:
# THIS STEP IS CRITICAL. If you don't put again in the stdout the
# input data, following tools in the pipe chain wouldn't receive
# the data
write_to_stdout_by_file_ref(input_json)

Changelog
You can check Changelog.

Documentation
You can find the complete documentation at: Documentation

Authors
Powerful Pipes was made by 42Crunch Research Team:

jc42
cr0hn

License
Powerful Pipes is Open Source and available under the Apache 2.

Contributions
Contributions are very welcome. See CONTRIBUTING.md or skim existing tickets to see where you could help out.

Acknowledgements
Special thanks to Cesar Gallego for his mentoring in data processing, that inspired this project.
Project logo thanks to Pipe icons created by srip - Flaticon