GitLocker: The Coding Marketplace

Description:

sftptos3 0.1.1

Welcome to sftp_to_s3 Documentation
sftp_to_s3 is an Open Source solution to ingest large amount of data from SFTP server to AWS S3.

How it Work
This solution assume that the SFTP server folder structure has to have partition strategy. For example, the SFTP server folder structure is like this:
/data/2022-01-01/many-folder-many-files...
/data/2022-01-02/many-folder-many-files...
/data/2022-01-03/many-folder-many-files...
/data/...
Once a partition of files are created on SFTP server, it should not be changed to frequently. Files in each partition are grouped into many batches to ensure that the total amount of files and total file size has a reasonable boundary. A coordinator will be responsible to detect the delta of partitions on SFTP and AWS S3, and then group files into batches. Then the batch data will be sent to a cluster of worker to download in parallel if it fits the SFTP server’s network bandwidth.
The coordinator and worker can be deployed as either AWS Lambda function or AWS Batch job. Also this solution uses DynamoDB to track the status of each partition and each batch to ensure the data integrity.
Please read the following links to learn more:

Architecture design
Sample project using this library
Sample script to generate fake data for integration test
Sample script to run the downloader

Partition Tracker Data:
{
"n_batches": 2, # this partition has 2 batches to download
"n_succeeded_batches": 2, # number of succeeded batch
"sftp_dir": "/home/username/part0001", # SFTP directory of this partition
"s3_dir": "s3://bucket/projects/sftp_to_s3/download/part1", # where you store the downloads
"n_files": 6, # total number of files in this partition
"total_size": 5430500 # total size of files in this partition
}
Request Tracker Data:
# this is the first batch
{
"partition_key": "part1", # partition key
"request_id": "request1", # request id / batch id
"sftp_dir": "/home/username/part1", # SFTP directory of this partition
"s3_dir": "s3://bucket/projects/sftp_to_s3/download/part1", # where you store the downloads
"s3uri_batch": "s3://bucket/projects/sftp_to_s3/requests/part1/request1.json", # where you store the batch data
"n_files": "3", # total number of files in this batch
"total_size": 3442228 # 3.4 MB # total size of files in this batch
}
Batch data JSON file:
# this is the sample content of the batch data
# it is the "s3uri_batch" field in Request Tracker Data
{
"dir_root": "/home/username/part1", # SFTP directory of this partition
"s3uri_root": "s3://bucket/projects/sftp_to_s3/download/part1", # where you store the downloads
"files": [ # list of files to download
"/home/username/part1/file0001.txt",
"/home/username/part1/file0002.txt",
"/home/username/part1/file0003.txt"
],
"n_files": "3", # total number of files in this batch
"total_size": 3442228 # 3.4 MB # total size of files in this batch
}

Install
sftp_to_s3 is released on PyPI, so all you need is:
$ pip install sftp_to_s3
To upgrade to latest version:
$ pip install --upgrade sftp_to_s3