Description:

link.parallel 1.1

link.parallel is a database agnostic query system.
See documentation for more informations.











Installation
pip install link.parallel


Features


parallel loops interface with IPython and multiprocessing drivers
Map/Reduce middleware




Examples
Create your map/reduce functions:
from b3j0f.task import register_task


@register_task('mymap')
def mymap(mapper, item):
if item['i'] < 5:
mapper.emit('i_lt_5', item)

elif item['i'] > 5:
mapper.emit('i_gt_5', item)

else:
mapper.emit('i_eq_5', item)


@register_task('myreduce')
def myreduce(key, values):
return (key, len(values))
Get input data and pass it to the middleware:
from link.middleware.core import Middleware

mapreduce = Middleware.get_middleware_by_uri(
'mapreduce+ipython:///test/classify?mapcb=mymap&reducecb=myreduce'
)

items = # load items
result = dict(mapreduce(items))

print(result)


Donating

License

For personal and professional use. You cannot resell or redistribute these repositories in their original state.

Share

• Share on Facebook
• Share on Twitter