Pyshrimp 0.8.0

Description:

pyshrimp 0.8.0

PyShrimp

PyShrimp is combination of utilities designed to support easy creation of small python scripts instead of getting your
hands dirty with shell scripting languages like bash.
When trying to write simple script to hack some ad-hoc task it's easy to end up with following choices:

use bash - it's easy to write simple script which invokes some processes. But then it's hard to do something more
advanced (lacking good support for types, lacking support for arrays...)
use python - is far better in handing different types, have good support for various collections, can easily be
expanded by installing dependencies from pip. But then there is painful overhead in setting up such script
(handling virtualenv to separate dependencies, or using one big env for all scripts) and doing more complex process
invocations beyond simple subprocess.check_output(...) is also far from ideal.

The PyShrimp aims at solving this - it's purpose is to remove barriers so developers can use python for simple
shell-scripting purpose. No more over-grown bash scripts, no more pain with env setup or subprocess handling!
Note: the features provided are not expected to replace poetry and other tools like that - if you can afford the
complexity then it's probably better to use those ;). But for single-file scripts it should be way easier to go with
PyShrimp.
TL;DR: Showcase
The rest of README explains features in detail. For those impatient there is quick example script - the code
is often worth 1000 words (although it covers only small subset of features):
#!/usr/bin/env pyshrimp
# $opts: magic,devloop
# $requires: click==7.0,requests==2.22.0
import requests
from click import style

from pyshrimp import log, as_dot_dict

def slog(message, **kwargs):
log(style(message, **kwargs))

slog('Loading issues...')

res = as_dot_dict(
requests.get(
'https://jira.atlassian.com/rest/api/2/search', params={
'jql': 'created > -1d',
'maxResults': '5'
}
).json(),
'res'
)

slog(f'Issues ({res.total} in total):', fg='green', bold=True)

for issue in res.issues:
line_color = 'red' if issue.fields.issuetype.name == 'Bug' else None
f = issue.fields
slog(f' * [{issue.key}] [{f.issuetype.name}] [{f.status.name}] {f.summary}', fg=line_color)

To run this script you need to have the PyShrimp installed:
pip install pyshrimp
chmod +x thescript.py
./thescript.py

PyShrimp will automatically initialize new virtual environment, install dependencies declared in file header,
and execute the script with devloop:

Features
Virtual env setup - $requires
PyShrimp parses script header and looks for the # $requires: ... lines.
Each such line can contain one or more pip-style requirements, for example:
#!/usr/bin/env pyshrimp
# $requires: click==7.0,requests==2.22.0
# $requires: PyYAML==5.3.1

Before script execution PyShrimp will create dedicated virtual env in ~/.cache/pyshrimp/{hash}/
where {hash} is hash value from the requirements. The environment is created only once - subsequent
runs of script will use already existing environment. Scripts with exactly the same dependencies will
re-use single virtual environment.
The $requirements_file can be used to load requirements list from external file:
#!/usr/bin/env pyshrimp
# $requirements_file: requirements.txt

It is also possible to use alternative keywords: $requirementsFile, $requires_file, $requiresFile.
Creation of new script
To quickly create new script just run the new command:
pyshrimp new my-new-script.py

The file created will contain skeleton of script. Script will have the executable mode set already.
Script startup - run
The run function makes it easy to run the main function with some extra capabilities.
The default behavior is to set up logging and run the script main function.
In addition to this it's possible to enable additional tweaks:

devloop - runs script in loop - PyShrimp will re-execute script automatically after it changes
elevate - ensures that script is running as root - uses sudo to elevate permissions

Example script with both features enabled:
#!/usr/bin/env pyshrimp
from pyshrimp import run, log

def main():
log('Hello world!')

run(main, devloop=True, elevate=True)

Power of magic
There is alternative 'magic' way to run script setup (logging configuration) and achieve the bonus features
like devloop and elevate. This is called magic as the PyShrimp no longer will only set up virtualenv but also
will wrap the code with extra "magical" setup.
The following script will have exactly the same behavior as previously presented with run method:
#!/usr/bin/env pyshrimp
# $opts: magic,devloop,elevate
from pyshrimp import log

log('Hello world!')

As you can see there is less "boilerplate" code when using this approach.
There is however one downside - the code run with just python will behave differently - the magic options will not
activate and also the logging will be not configured. This can be surprising when running script with debugger so
use the magic wisely ;).
Useful utilities
There are few useful utilities provided:

as_dot_dict - creates dictionary wrapper with support for property-like access
to the values: as_dot_dict(d).some_key.some_list[1].some_value
unwrap_dot_dict - un-wraps DotDicts back into the raw dict/list
ls, glob_ls - lists files and directories
write_to_file, read_file, read_file_bin - file content manipulation
chmod_set, chmod_unset - sets/unsets file mode bits
acquire_file_lock, FileBasedLock - handles file based locking
re_match_all - runs regular expression matching across the list and returns selected
group from the matched elements
in_background - runs function in background thread pool
StringWrapper - provides few methods especially useful for parsing process output
parse_table - parses table-like output into ParsedTable
create_regex_splitter - creates regex_splitter - useful to handle unusual table/column-like output
wait_until, wait_until_gen - handles waiting for some result with timeout using periodic polling

You can see example usage in examples and also in tests.
Easily run programs - simple subprocess helper
The subprocess helper simplifies the task of running processes and handling the results.
from pyshrimp import run_process
print(
run_process('echo -n 123 | wc -c', run_in_shell=True).raise_if_not_ok().standard_output
)

More advanced subprocess helper
The cmd and shell_cmd will produce Command object which can be executed with extra params:
from pyshrimp import shell_cmd
wc_c = shell_cmd('echo -n "$1" | wc -c', check=True)
print(wc_c('123456789').standard_output.strip()) // 9
print(wc_c.exec('123').standard_output.strip()) // 3

It is worth noting here that standard_output and error_output
are wrapped with StringWrapper to ease output parsing.
Pipeline support
Motivation
The nice property about bash scripts is how easy it is to executed process and pass the output between processes. It's
all possible in python but the overhead is really too big to use in small scripts. It's more convenient to fallback into
subprocess with partial shell script than glue together the processes directly in python code.
Solution
The ExecutionPipeline is designed to address those concerns. You can easily stitch together few processes and
functions together to process the input and output just like in shell scripts.
There are two approaches - standard object-oriented python code and more shell like pipeline syntax.
Object oriented usage
from pyshrimp import ExecutionPipeline, cmd
p = ExecutionPipeline()
# feed pipeline with text input
p.attach_text('Hello world!')
# run the wc command
p.attach(cmd('wc'))
# run awk, using the shell wrapper
p.attach("awk '{print $3}'")
# process the output with function - pad with zeros
p.attach_function(
lambda stdin: f'{int(stdin.strip()):05}'
)
# close pipeline and collect output
res = p.close().stdout
print(res) # 00012

Shell-like pipe syntax
The shell-like syntax is "abusing" the python feature which allows overriding behavior of binary or operator. It's not
very elegant (could be confusing for people) but on the other hand this fits nicely the purpose - being easy replacement
of shell pipes.
from pyshrimp import PIPE, PIPE_END_STDOUT, cmd
res = (
# feed pipeline with text input
PIPE.text('Hello world!')
# run the wc command
| cmd('wc')
# run awk, using the shell wrapper
| "awk '{print $3}'"
# process the output with function - pad with zeros
| (lambda stdin: f'{int(stdin.strip()):05}')
# close pipeline and collect output
| PIPE_END_STDOUT
)
print(res) # 00012

Mixed approach
You can mix the shell-like and object-oriented syntax (if you have good reason to do so ;)). Under the cover
the ExecutionPipeline is being used even in the shell-like syntax.
from pyshrimp import PIPE, cmd
p = (
# feed pipeline with text input
PIPE.text('Hello world!')
# run the wc command
| cmd('wc')
# run awk, using the shell wrapper
| "awk '{print $3}'"
)
# process the output with function - pad with zeros
p.attach_function(lambda stdin: f'{int(stdin.strip()):05}')
# close pipeline and collect output
res = p.close().stdout
print(res) # 00012

Limitations
Things obviously missing in current version that you should be aware of:

Limited error handling support
Only text streams are officially supported, with UTF-8 hardcoded
Only connection of standard output is supported - error output behavior is undetermined (most likely flows to stderr)

Those limitations can be addressed in future (if there is enough demand and willingness to introduce the change).
Troubleshooting
Diagnosing virtual env setup failure
You can set PYSHRIMP_LOG environment variable to 1 this will instruct boostrap code to
produce diagnostic messages:
% PYSHRIMP_LOG=1 ./show_recently_created_issues.py a b
[PyShrimp:bootstrap] INFO: target: ./show_recently_created_issues.py
[PyShrimp:bootstrap] INFO: args: ['a', 'b']
[PyShrimp:bootstrap] INFO: Using requirements d82e6efbb5ea5ba895b6fe103b4c50bf3ac75eb3: [
'pyshrimp', 'click==7.0'
]
[PyShrimp:bootstrap] INFO: Executing the script: [
'~/.cache/pyshrimp/virtual_envs/d82e6e.../bin/python', '-u',
'-m', 'pyshrimp._internal.wrapper.magicwrapper',
'--', './show_recently_created_issues.py', 'a', 'b'
]

Hello world

Supported environment
This project was developed on Ubuntu Linux. It should work with any linux system, but I can imagine the tests
failing in case some system binaries are missing.
The project is also tested on macOS before new version is released.
Some parts for sure will not work on Windows (e.g. shell wrapping is depending on bash).
The author does not have plans to introduce support for Windows but contributions are welcome ;).
License
The project is licensed under MIT License with exceptions listed below.
Project license exceptions:

The files in doc/assets/img/logo and docs/assets/img/logo directory
are licensed under CC BY-SA 3.0 license.

Contributions
Feel free to contribute to this project - I'll do my best to review and accept contributions.
Please include at least some happy-path tests for your changes.
Imaginary Q&A
Q: Shouldn't this project be separated into few ones (e.g. pipelines, commands, script bootstrap)?
A: Probably yes. And maybe it will be split in the future. But for now it's more convenient to manage single project.
Credits

The logo was created using "Shrimp" icon
created by "elmars" and published
under CC BY-SA 3.0 license.

Files In This Product: