parallel-manager 0.1.0

Description:

parallelmanager 0.1.0

parallel_manager

A simple shell command manager. Used for running parallel simulations.
How-to
First install the package with pip:
pip install parallel_manager

To run the manager, we will need to wrap the manager initialization and request creations all in a async function like this:
from parallel_manager.manager import BaseShellManager
from parallel_manager.workerGroup import ShellWorkerGroup
import logging
import asyncio

async def Main():
# Init the manager
simpleShellWorkergroup = ShellWorkerGroup("simpleShellWorkergroup",
logging.getLogger(),
"./log",
10)
simpleShellManager = BaseShellManager("simpleShellManager")
simpleShellManager.add_workergroup("shell", simpleShellWorkergroup)
await simpleShellManager.init()

# Adding tasks
# Here we just run 100 echo
for i in range(100):
simpleShellManager.add_shell_request(f"echoing loop-{i}", f"echo {i}")

# Wait for the manager to finish
await simpleShellManager.done()

Then we simply use asyncio.run() to run the above function:
asyncio.run(Main())

The manager will run at most 10 processes at a given time and will terminate when all 100 requests are done.
Full example is at here
Structure

Worker

Receive request from queue and process it


WorkerGroup

Manage workers of same class
Collect basic stats from workers


Manager

Might host various workergroups with different worker type
Support various ways to add tasks
Control workergroups/workers
Various middleware/plugin to the BaseManager to implement other features like http web monitoring and task submission



TODOs
Short-term todos

Manager statistics plugin
Manager summary when all tasks are done
Timeout support
Status monitor, progress monitor
Dump failed task/able to rerun
Dump unfinished tasks when script terminates

Long-term todos

Thread-mode support (completely non-blocking)

Use a separate thread to run the workers
Allow to submit task at any time


Http monitor
Interactive control for status monitoring and changing configuration dynamically, treat worker as a server that can submit task to, like a local slurm control
PythonFunctionWorker class (execute python function calls) and other worker class as well
Standardize plugin format for each class

For instance, http server monitor could be a plugin that can be added to any manager class.