python-worker 2.2.4

Description:

pythonworker 2.2.4

python-worker


Description
A package to simplify the thread declaration directly either by using decorator or pass it through function. It also allows you to stop the running thread (worker) from any layer

Installation
pip install python-worker


Changelogs

v1.8:

Refactoring codes
flexible worker declaration


v1.9:

Added Asynchronous Worker for coroutine function using @async_worker decorator


v1.10:

Added overload typehints for worker and async_worker
Added restart feature for worker


v2.0:

Added process worker to enable run your function in different GIL (Global Interpreter Lock) which could give you a performance boost


v2.2:

Added async_process worker to enable run your async function / coroutine in different GIL (Global Interpreter Lock) which could give you a performance boost




Basic Guide
@worker will define a function as a thread object once it run
import time
from worker import worker

@worker
def go(n, sleepDur):
for i in range(n):
time.sleep(sleepDur)
print('done')

go(100, 0.1)

The function go will be running as a thread

Asynchronous Guide
Well, if you have a coroutine function you can use async_worker instead
import asyncio
from worker import async_worker

@async_worker
async def go():
print("this is inside coroutine!")
for i in range(10):
time.sleep(0.5)
print(i)
print("done!")
return "result!"

go_worker = asyncio.run(go())

or run it as a process
import asyncio
from worker import async_process

@async_process
async def go():
print("this is inside coroutine!")
for i in range(10):
time.sleep(0.5)
print(i)
print("done!")
return "result!"

go_worker = asyncio.run(go())


Process Guide
A new feature called process is simply putting your worker on different GIL (Global Interpreter Lock) which could give you a performance boost.
It's implementing multiprocessing instead of multithreading which at this stage is achieving the true form of parallelism which is run in different environment with your function call environment
import time
import os
from worker import process

@process
def go(n, sleepDur):
for i in range(n):
time.sleep(sleepDur)
print('done')

go(100, 0.1)

your function go will run in different process.
To check it, you can just print out the os.getpid()
import os
from worker import worker, process

@worker(multiproc=True)
def run_in_new_process_from_worker(parent_pid):
print(f"from {parent_pid} running in a new process {os.getpid()} - from worker.mutliproc==True")
return "return from process"

@process
def run_in_new_process(parent_pid):
print(f"from {parent_pid} running in a new process {os.getpid()} - from process")
return "return from process"

@worker
def run_in_new_thread(parent_pid):
print(f"from {parent_pid} running in a new thread {os.getpid()} - from worker.multiproc==False")
return "return from thread"


print(f"this is on main thread {os.getpid()}")

run_in_new_process_from_worker(os.getpid())
run_in_new_process(os.getpid())
run_in_new_thread(os.getpid())


then run the script
this is on main thread 29535
from 29535 running in a new process 29537 - from worker.mutliproc==True
from 29535 running in a new thread 29535 - from worker.multiproc==False
from 29535 running in a new process 29538 - from process

you can see the different of process id between running in a new process and thread

Additional Guides
Kill / Stop / Abort the running worker
You can abort some workers, all workers or even all threads..
Abort specific workers
import time
from worker import worker, abort_worker

@worker
def go4(n=10):
for i in range(n):
time.sleep(1)

go4_worker = go4(10)
time.sleep(3)
abort_worker(go4_worker)

or just abort it from the instance
go4_worker.abort()

Abort all workers (this only abort worker threads only)
from worker import abort_all_worker

abort_all_worker()

Abort all threads (it will abort both all worker and non-worker threads)
from worker import abort_all_thread

abort_all_thread()


Run undefined @worker function
import time
from worker import run_as_Worker

def go(n):
...

go_worker = run_as_Worker(target=go, args=(10,))


Get Return Value
How to get the return of threaded function ?
@worker
def go(n):
time.sleep(n)
return "done"

go_worker = go(10)

# this will await the worker to finished and return the value

go_result = go_worker.await

# You can also use this if it's finished, dont have to await

go_result = go_worker.ret

Check/Monitor All Workers
from worker import ThreadWorkerManager

## all created workers
ThreadWorkerManager.list()

## All active/running workers only
ThreadWorkerManager.list(active_only=True)

it will return the information
>>> ThreadWorkerManager.list()
==============================================================
ID |Name |Active|Address | WorkTime (s)
==============================================================
0 |worker |True |0x7fdf1a977af0 | 4.97
1 |worker1 |True |0x7fdf1a73d640 | 4.07
2 |worker2 |True |0x7fdf1a73d9d0 | 3.83
3 |worker3 |True |0x7fdf1a73dd00 | 3.62
4 |worker4 |True |0x7fdf1a74b070 | 3.38
==============================================================


Python Interactive Shell - Keyboard Interrupt (CTRL+C)
When you run your scripts on interactive mode
python -i myScript.py

you could add an abort handler with keyboard interrupt to abort your thread.
Inside myScript.py
ThreadWorkerManager.enableKeyboardInterrupt() allows you to abort your running workers.
from worker import worker, ThreadWorkerManager


# enabling abort handler for worker into keyboard interrupt (CTRL+C)

ThreadWorkerManager.enableKeyboardInterrupt()

You could also activate exit thread which triggered by pressing the CTRL+Z. This also added an abort handler for worker into keyboard interrupt (CTRL+C).
ThreadWorkerManager.disableKeyboardInterrupt(enable_exit_thread=True)

Disabling abort handler for worker into keyboard interrupt (CTRL+C).
ThreadWorkerManager.disableKeyboardInterrupt()

Check handler status.
ThreadWorkerManager.keyboard_interrupt_handler_status

You also can choose which workers are allowed to be aborted on keyboard interrupt
Inside myScript.py
from worker import worker, ThreadWorkerManager

@worker("Uninterrupted", on_abort=lambda: print("ITS GREAT"), keyboard_interrupt=False)
def go_not_interrupted():
i = 0
while i < 1e3/2:
i += 10
print(i,"go_not_interrupted")
time.sleep(0.001)
return i

@worker("Interrupted", on_abort=lambda: print("ITS GREAT"), keyboard_interrupt=True)
def go_interrupted():
i = 0
while i < 1e3/2:
i += 10
print(i,"go_interrupted")
time.sleep(0.001)
return i

ThreadWorkerManagerManager.enableKeyboardInterrupt()
go_not_interrupted()
go_interrupted()

run in your terminal
python -i myScript.py

press CTRL+C while the process is running and see the results.