GitLocker: The Coding Marketplace

Description:

pytaskjulia 0.4.0

pytask-julia

Run Julia scripts with pytask.
Installation
pytask-julia is available on PyPI and
Anaconda.org. Install it with
$ pip install pytask-julia

# or

$ conda install -c conda-forge pytask-julia

You also need to have Julia installed and julia on your command line. Test it by
typing the following on the command line
$ julia -h

If an error is shown instead of a help page, you can install Julia on Unix systems with
$ conda install -c conda-forge julia

or choose one of the installers on this page.
Usage
To create a task which runs a Julia script, define a task function with the
@pytask.mark.julia decorator. The script keyword provides a path relative to the
task module to the Julia script.
import pytask
from pathlib import Path
from pytask import task

@task(kwargs={"path": Path("out.csv")})
@pytask.mark.julia(script="script.jl")
def task_run_jl_script():
pass

If you are wondering why the function body is empty, know that pytask-julia replaces the
body with a predefined internal function. See the section on implementation details for
more information.
Dependencies and Products
Dependencies and products can be added as usual. Read this
tutorial.
For example, with the @pytask.task decorator as shown before. (The choice of the kwarg
name, here path, is arbitrary.)
Accessing dependencies and products in the script
To access the paths of dependencies and products in the script, pytask-julia stores the
information by default in a .json file. The path to this file is passed as a
positional argument to the script. Inside the script, you can read the information.
import JSON

path_to_json = ARGS[1] # Contains the path to the .json file.

config = JSON.parse(read(path_to_json, String)) # A dictionary.

config["path"] # Is the path to the output file "../out.csv".

The .json file is stored in the same folder as the task in a .pytask directory.
To parse the JSON file, you need to install
JSON.jl.
You can also pass any other information to your script by using the @task decorator.
@task(kwargs={"path": Path("out.csv"), "number": 1})
@pytask.mark.julia(script="script.jl")
def task_run_jl_script():
pass

and inside the script use
config["number"] # Is 1.

Debugging
In case a task throws an error, you might want to execute the script independently from
pytask. After a failed execution, you see the command which executed the Julia script in
the report of the task. It looks roughly like this
$ julia <options> -- script.jl <path-to>/.pytask/task_py_task_example.json

Managing Julia environments
Julia has support for environments to execute your tasks via Pkg.jl which is explained
here.
pytask-julia allows you define a default environment via your
pytask configuration file.
Use the julia_project key to define a path to your configuration file to point to your
environment.
Probably your environment files Manifest.toml and Project.toml reside at the root of
your project folder as well as your pytask configuration file. Then, the content will
look like this.
[tool.pytask.ini_options]
julia_project = "."

You can also define environments for each task which will overwrite any other default
with the project keyword argument. Pass a path to the task module.
@task(kwargs={"path": Path("out.csv")})
@pytask.mark.julia(script="script.jl", project=".")
def task_run_jl_script():
pass

Command Line Options
Command line options can be pass via the options keyword argument.
@task(kwargs={"path": Path("out.csv")})
@pytask.mark.julia(script="script.jl", options=["--threads", "2"])
def task_run_jl_script():
pass

This example will execute the script using to threads.
Repeating tasks with different scripts or inputs
You can also repeat the execution of tasks, meaning executing multiple Julia scripts or
passing different command line arguments to the same Julia script.
The following task executes two Julia scripts, script_1.jl and script_2.jl, which
produce different outputs.
for i in range(2):

@task(kwargs={"path": Path(f"out_{i}.csv")})
@pytask.mark.julia(script=f"script_{i}.jl")
def task_execute_julia_script():
pass

If you want to pass different inputs to the same Julia script, pass these arguments with
the kwargs keyword of the @task decorator.
for i in range(2):

@task(kwargs={"path": Path(f"out_{i}.csv"), "i": i})
@pytask.mark.julia(script="script.jl")
def task_execute_julia_script():
pass

and inside the task access the argument i with
import JSON

path_to_json = ARGS[1] # Contains the path to the .json file.

config = JSON.parse(read(path_to_json, String)) # A dictionary.

config["path"] # Is the path to the output file "../output_{i}.csv".

config["i"] # Is the number.

Serializers
You can also serialize your data with any other tool you like. By default, pytask-julia
also supports YAML (if PyYaml is installed).
Use the serializer keyword arguments of the @pytask.mark.julia decorator with
@pytask.mark.julia(script="script.jl", serializer="yaml")
def task_example():
...

And in your Julia script use
import YAML
config = YAML.load_file(ARGS[1])

Note that the YAML package needs to be installed.
If you need a custom serializer, you can also provide any callable to serializer which
transforms data to a string. Use suffix to set the correct file ending.
Here is a replication of the JSON example.
import json

@pytask.mark.julia(script="script.jl", serializer=json.dumps, suffix=".json")
def task_example():
...

Configuration
You can influence the default behavior of pytask-julia with some configuration values.
Place them into a pyproject.toml file.
julia_serializer
Use this option to change the default serializer.
[tool.pytask.ini_options]
julia_serializer = "json"

julia_suffix
Use this option to set the default suffix of the file which contains serialized paths to
dependencies and products and more.
[tool.pytask.ini_options]
julia_suffix = ".json"

julia_options
Use this option to set default options for each task which are separated by whitespace.
[tool.pytask.ini_options]
julia_options = ["--threads", 2]

julia_project
Use this option to set a default environment for each task. If your environment with
Manifest.toml and Project.toml is defined in the same directory as the configuration
file pyproject.toml, just use a dot.
[tool.pytask.ini_options]
julia_project = "."

If the environment files were in a folder next to the configuration file called
environment use
[tool.pytask.ini_options]
julia_project = "environment"

Changes
Consult the release notes to find out about what is new.
Development
After setting up the environment, enter
$ julia
$ ]
$ pkg> activate .
$ pkg> instantiate

to set up the Julia environment.