avroconvert 0.1.1

Description:

avroconvert 0.1.1

avroconvert



Utility to convert avro files to csv, json and parquet formats


Installation


Using pypi
pip install avroconvert

Using git:
git clone https://github.com/shrinivdeshmukh/avroconvert

make install



Usage


Using CLI
CLI can be used to interact with the tool. As the first argument, the source has to be passed. The source can be gs (google cloud storage bucket), s3 (amazon s3 bucket) or fs (local filesystem)
To read from cloud bucket (google cloud or amazon s3):
google cloud storage example:
avroconvert gs -b <BUCKET_NAME> -f <FORMAT> -o <OUTPUT_FOLDER>

amazon s3 example:
avroconvert s3 -b <BUCKET_NAME> -f <FORMAT> -o <OUTPUT_FOLDER>

The tool reads all avro files from the bucket specified by the -b parameter, converts them to the format specified by the -f parameter, and writes the output format files to the output folder specified by the -o parameter with the above command.
The cli accepts a few additional parameters to authenticate the tool with cloud providers. These parameters are only required if you haven't already been authenticated.
For google cloud, we have --auth-file:
avroconvert gs -b <BUCKET_NAME> -f <FORMAT> -o <OUTPUT_FOLDER> --auth-file <SERVICE_ACCOUNT_FILE_PATH>.json (or .p12)

For amazon s3, we have --access-key, --secret-key, --session-token:
avroconvert s3 -b <BUCKET_NAME> -f <FORMAT> -o <OUTPUT_FOLDER> --access-key <AWS_ACCESS_KEY_ID> --secret-key <AWS_SECRET_ACCESS_KEY> --session-token <AWS_SESSION_TOKEN>

To read from local filesystem
avroconvert fs -i <INPUT_DATA_FOLDER> -o <OUTPUT_FOLDER> -f <OUTPUT_FORMAT>

The tool reads all avro files from the input folder specified by the -i parameter, converts them to the format specified by the -f parameter, and writes the output format files to the output folder specified by the -o parameter with the above command.
Output folder structure
The tool replicates the cloud bucket's or local filesystem's directory structure. For example, suppose the output format is parquet and cloud bucket (or local filesystem) has the following structure:
BUCKET
├── 2021-06-17
│ └── file1.avro
│ └── file2.avro
│
├── 2021-06-16
│ └── data
│ └── file3.avro
│ └── file4.avro


the output files will then be saved as:
OUTPUT_FOLDER
├── 2021-06-17
│ └── file1.parquet
│ └── file2.parquet
│
├── 2021-06-16
│ └── data
│ └── file3.parquet
│ └── file4.parquet


Filter files to read
A parameter called -p or —-prefix can be passed as well. All three data sources, gs, s3, and fs, share this parameter. Only files with names that begin with the specified prefix will be read; all other files will be filtered out.
google cloud example with -p:
avroconvert gs -b <BUCKET_NAME> -f <FORMAT> -o <OUTPUT_FOLDER> -p 2021-06-17/file

amazon s3 example with -p:
avroconvert s3 -b <BUCKET_NAME> -f <FORMAT> -o <OUTPUT_FOLDER> -p 2021-06-17/file

local filesystem example with -p:
avroconvert fs -i <INPUT_DATA_FOLDER> -o <OUTPUT_FOLDER> -f <OUTPUT_FORMAT> -p 2021-06-17/file

Using the API in code
from avroconvert import Execute

# for amazon s3 storage bucket reader
output = Execute(source='gs', bucket='<BUCKET_NAME>, dst_format='parquet', auth_file='<SERVICE_ACCOUNT.json>',
outfolder='OUTPUT_FOLDER', access_key='<AWS ACCESS KEY>', secret_key='<AWS SECRET KEY>',
session_token='<AWS SESSION TOKEN>(if any)', bucket='<S3 BUCKET>', prefix='<FILE PREFIX>').run()

# google storage bucket reader
output = Execute(source='gs', bucket='<BUCKET_NAME>, dst_format='parquet', auth_file='<SERVICE_ACCOUNT.json>',
outfolder='OUTPUT_FOLDER').run()

# Local file system reader
output = Execute(source='fs', bucket='<LOCAL_FOLDER NAME> dst_format='parquet', outfolder='OUTPUT_FOLDER').run()


For more details on using the API, please visit readthedocs


Credits


This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.