Description:

ctoddpythonlibminio 1.0.2

Christopher H. Todd's PROJECT_STRING_NAME
The PROJECT_GIT_NAME project is responsible for ...
The library ...
Table of Contents

Dependencies
Libraries
Example Scripts
Notes
TODO

Dependencies
Python Packages



Libraries
minio_bucket_helpers.py
This library is used to interact with Minio object storage. Will handle functions used to interact with buckets (creating, downloading, finding, etc)
Functions:
def get_buckets(minio_client):
"""
Purpose:
Get a list of buckets that exist in the Minio Client
Args:
minio_client (minio client Obj): Client obj connection to Minio
Returns:
buckets (List of Bucket Objs): List of Bucket OBJs in Minio
"""

def get_bucket_names(minio_client):
"""
Purpose:
Get a list of buckets that exist in the Minio Client
Args:
minio_client (minio client Obj): Client obj connection to Minio
Returns:
bucket_names (List of Strings): List of Buckets in Minio
"""


def create_bucket(minio_client, bucket_name):
"""
Purpose:
Create a specified Bucket by name
Args:
minio_client (minio client Obj): Client obj connection to Minio
bucket_name (String): Name of bucket to create
Returns:
N/A
"""

def delete_bucket(minio_client, bucket_name):
"""
Purpose:
Delete a specified Bucket by name
Args:
minio_client (minio client Obj): Client obj connection to Minio
bucket_name (String): Name of bucket to delete
Returns:
N/A
"""

minio_client.py
MinioClient Class for interacting with minio object store. Objects will be created connected to Minio
Classes:
class MinioClient(object):
"""
MinioClient Class. Class objects will hold connection to the
Minio service and can be used to interact with buckets and objects
"""

minio_connection_helpers.py
This library is used to interact with Minio object storage. Functions establish a connection to the Minio service that can be used to interact with the service and pass to the other helper functions
Functions:
def connect_to_minio(minio_url, access_key=None, secret_key=None, secure=False):
"""
Purpose:
Connect to Minio and return the minio_client of minio lib
Args:
minio_url (String): URL of Minio
access_key (String): Access Key for Minio
secret_key (String): Secret Key for Minio
Returns:
minio_client (minio client Obj): Client obj connection to Minio
"""

def build_minio_url(minio_host, minio_port=9000):
"""
Purpose:
Create the Minio URL from host and port
Args:
minio_host (String): Host of Minio
minio_host (Int): Port of Minio (Defaults to 9000)
Returns:
minio_url (String): URL of Minio
"""

minio_exceptions.py
File for holding custom exception types that will be generated by the minio_helpers libraries
Exceptions:
class BucketAlreadyExists(Exception):
"""
Purpose:
The BucketAlreadyExists will be raised when attempting to create a bucket
that already exists
"""

class BucketDoesntExist(Exception):
"""
Purpose:
The BucketDoesntExist will be raised when attempting to delete a bucket
that doesn't exist in Minio
"""

class ObjectAlreadyExists(Exception):
"""
Purpose:
The ObjectAlreadyExists will be raised when attempting to create an object
that already exists
"""

class ObjectDoesntExist(Exception):
"""
Purpose:
The ObjectDoesntExist will be raised when attempting to pull an object
that doesn't exist in Minio
"""

class ObjectDecodingNotSupported(Exception):
"""
Purpose:
The ObjectDecodingNotSupported will be raised when attempting to decode a
filetype from Minio that is not yet supported (such as .avro)
"""

minio_general_helpers.py
This library is used to interact with Minio object storage.
Functions:
N/A
minio_object_helpers.py
Example executable Python scripts/modules for testing and interacting with the library. These show example use-cases for the libraries and can be used as templates for developing with the libraries or to use as one-off development efforts.
Functions:
def get_objects(minio_client, bucket_name):
"""
Purpose:
Get a list of objects that exist in the Minio Client
Args:
minio_client (minio client Obj): Client obj connection to Minio
bucket_name (String): Name of the bucket to get objects for
Returns:
objects (List of Object Objs): List of Object OBJs in Minio
"""

def get_object_names(minio_client, bucket_name):
"""
Purpose:
Get a list of objects that exist in the Minio Client
Args:
minio_client (minio client Obj): Client obj connection to Minio
bucket_name (String): Name of the bucket to get objects for
Returns:
object_names (List of Strings): List of Objects in Minio
"""

def is_object_in_bucket(minio_client, bucket_name, object_name):
"""
Purpose:
Check if Object exists in Bucket
Args:
minio_client (minio client Obj): Client obj connection to Minio
bucket_name (String): Name of the bucket to check for object
object_name (String): Name of object to check for in Minio
Returns:
object_exists (Boolean): Boolean if the object exists or not
"""

def get_object_stats(minio_client, bucket_name, object_name):
"""
Purpose:
Get Stats of the Object
Args:
minio_client (minio client Obj): Client obj connection to Minio
bucket_name (String): Name of the bucket to check for object
object_name (String): Name of object to get stats for in Minio
Returns:
object_stats (Dict):Dict of stats about the object
"""

def download_object_to_memory(minio_client, bucket_name, object_name, encoding="utf-8"):
"""
Purpose:
Download an Object from Mino into memory (if supported)
Args:
minio_client (minio client Obj): Client obj connection to Minio
bucket_name (String): Name of the bucket to get object from
filename (String): Location (And Path) of file to upload
object_name (String): Name of object to upload in Minio
Returns:
parsed_object (Obj, depending on extension): Object parsed from Minio from the
extension of the file. Current supported = .txt -> str, .json -> Dict/JSON
"""

def download_object_to_file(minio_client, bucket_name, object_name, filename=None):
"""
Purpose:
Download a file from Minio to local storage
Args:
minio_client (minio client Obj): Client obj connection to Minio
bucket_name (String): Name of the bucket to get object from
filename (String): Location (And Path) of file to upload
object_name (String): Name of object to upload in Minio
Returns:
N/A
"""

def upload_object(minio_client, bucket_name, filename, object_name=None):
"""
Purpose:
Uploading a local file to Minio
Args:
minio_client (minio client Obj): Client obj connection to Minio
bucket_name (String): Name of the bucket to get to upload object to
filename (String): Location (And Path) of file to upload
object_name (String): Name of object to upload in Minio
Returns:
N/A
"""

def delete_object(minio_client, bucket_name, object_name):
"""
Purpose:
Delete a specified Object by name
Args:
minio_client (minio client Obj): Client obj connection to Minio
object_name (String): Name of object in Minio to delete
Returns:
N/A
"""

Example Scripts
Example executable Python scripts/modules for testing and interacting with the library. These show example use-cases for the libraries and can be used as templates for developing with the libraries or to use as one-off development efforts.
connect_to_minio.py
Purpose:
Connecting to Minio

Steps:
- Connect to Minio

function call:python3 connect_to_minio {--access-key=access_key} \
{--secret-key=secret_key} {--minio-host=minio_host} {--minio-port=minio_port}

create_bucket_in_minio.py
Purpose:
Create Bucket in Minio

Steps:
- Connect to Minio
- Create a Bucket

function call:python3 create_bucket_in_minio.py {--access-key=access_key} \
{--secret-key=secret_key} {--minio-host=minio_host} {--minio-port=minio_port} \
{--bucket-name=bucket_name}

delete_bucket_from_minio.py
Purpose:
Delete Bucket from Minio

Steps:
- Connect to Minio
- Delete a Bucket

function call:python3 delete_bucket_from_minio.py {--access-key=access_key} \
{--secret-key=secret_key} {--minio-host=minio_host} {--minio-port=minio_port} \
{--bucket-name=bucket_name}

get_objects_from_bucket.py
Purpose:
Get all Objects (Or a Specific Object) From a Bucket in Minio

Steps:
- Connect to Minio
- Get the Bucket Obj
- Get objects from Minio and store locally

function call:python3 get_objects_from_Bucket.py {--access-key=access_key} \
{--secret-key=secret_key} {--minio-host=minio_host} {--minio-port=minio_port} \
{--bucket-name=bucket_name} {--object-name=object_name}

Notes

Relies on f-string notation, which is limited to Python3.6. A refactor to remove these could allow for development with Python3.0.x through 3.5.x

TODO

Unittest framework in place, but lacking tests

License

For personal and professional use. You cannot resell or redistribute these repositories in their original state.

Share

• Share on Facebook
• Share on Twitter