GitLocker: The Coding Marketplace

Description:

pclip 0.2.8

Clip Ship Planet CLI addon

Note: The Clips API has been deprecated - and will no longer be supporting the standalone Clip and Ship service. Deprecation means that there is intention to remove the service at some point; it DOES NOT mean end-of-life for the service, yet. (Deprecate and end-of-life are two distinct terms.) There should be a plan to replace Clips API with new pre-processing functionality, but no new announcement or timetable for the launch of the new service and the removal (end-of-life) for Clips API has been made.
Planet's Clip API was a compute API designed to allowed users to clip the images to their area of interest. This would save them time in preprocessing and also allow the user to save on their area quota which might have restrictions. Based on Planet's Education and Research Program this quota is set at 10,000 square kilometers a month, which means saving up on quota is very useful. The discussion also led to an important clarification that users are in fact charged only for the area downloaded post clip if using the clip operation and hence this tool. This tool takes a sequential approach from activation to generating a clip request for multiple images activated and then processing the download tokens to actually download the clipped image files. The tool also consists of a sort function which allows the user to extract the files and sort them by type and deleting the original files to save on space.
Installation
To install the Clip-Ship-Planet-CLI you can simply perform the following action with Linux(Tested on Ubuntu 16):
git clone https://github.com/samapriya/Clip-Ship-Planet-CLI.git
cd Clip-Ship-Planet-CLI && pip install -r requirements.txt

On a windows as well as a linux machine, installation is an optional step; the application can also be run directly by executing pclip.py script. The advantage of having it installed is being able to execute ppipe as any command line tool. I recommend installation within virtual environment but you can also install it to system python and should not create any conflicts. To install on windows download the setup files as a zip package, unpack and run
python setup.py develop or python setup.py install

In a linux distribution
sudo python setup.py develop or sudo python setup.py install
pclip -h

Table of contents

Getting started
Usage examples

Planet Key
Planet Quota
AOI JSON
Activate or Check Asset
List IDs
Clipping with GeoJSON
Clipping with JSON
Downloading Clipped Imagery
Sorting

Getting started
As usual, to print help:
Planet Clip Tools CLI

positional arguments:
{ ,planetkey,aoijson,activate,aoiupdate,idlist,geojsonc,jsonc,downloadclips,sort}
-------------------------------------------
-----Choose from Planet Clip Tools-----
-------------------------------------------
planetkey Enter your planet API Key
quota Prints your quota details
aoijson Tool to convert KML, Shapefile,WKT,GeoJSON or Landsat
WRS PathRow file to AreaOfInterest.JSON file with
structured query for use with Planet API 1.0
activate Tool to query and/or activate Planet Assets
idlist Allows users to generate an id list for the selected
item and asset type for example item_asset=
PSOrthoTile analytic/PSScene3Band visual. This is used
with the clip tool
geojsonc Allows users to batch submit clipping request to the
Planet Clip API using geometry in geojson file
jsonc Allows users to batch submit clipping request to the
Planet Clip API using geometry in structured json
file. This is preferred because the structured JSON
allows the activate tool to stream line asset ids
being requested and to extract geometry from the same
file
downloadclips Allows users to batch download clipped assets post
computation using a directory path(Requires you to
first activate and run geojson or json tool)
sort Allows users to unzip downloaded files to new folder
and sorts into images and metadata

optional arguments:
-h, --help show this help message and exit

Usage examples
The tools have been designed to follow a sequential setup from activation, clip, download and even sort and includes steps that help resolve additional issues a user might face trying to download clipped area of interests instead of entire scenes. The system will ask you to enter your API key before the CLI starts(this will prompt you only once to change API key use the Planet Key tool).
Planet Key
This tool basically asks you to input your Planet API Key using a password prompt this is then used for all subsequent tools. Ites makes use of the Planet client and esentially executes planet init
usage: pclip planetkey [-h]

optional arguments:
-h, --help show this help message and exit

Planet Quota
This tool prints details on your existing quota and your area remaining
usage: pclip quota

optional arguments:
-h, --help show this help message and exit

AOI JSON
The aoijson tab within the toolset allows you to create filters and structure your existing input file to that which can be used with Planet's API. The tool requires inputs with start and end date, along with cloud cover. You can choose from multiple input files types such as KML, Zipped Shapefile, GeoJSON, WKT or even Landsat Tiles based on PathRow numbers. The geo option asks you to select existing files which will be converted into formatted JSON file called aoi.json. If using WRS as an option just type in the 6 digit PathRow combination and it will create a json file for you.
usage: pclip aoijson [-h] [--start START] [--end END] [--cloud CLOUD]
[--inputfile INPUTFILE] [--geo GEO] [--loc LOC]

optional arguments:
-h, --help show this help message and exit
--start START Start date in YYYY-MM-DD?
--end END End date in YYYY-MM-DD?
--cloud CLOUD Maximum Cloud Cover(0-1) representing 0-100
--inputfile INPUTFILE
Choose a kml/shapefile/geojson or WKT file for
AOI(KML/SHP/GJSON/WKT) or WRS (6 digit RowPath
Example: 023042)
--geo GEO map.geojson/aoi.kml/aoi.shp/aoi.wkt file
--loc LOC Location where aoi.json file is to be stored

As with the Planet-GEE-Pipeline-CLI the aoijson tool allows the user to bring any filetype of interest, which includes GEOJSON, WKT, KML or SHP file including but not limited to WRS rowpath setup and structures it to enable filtered query using Planet's data API. A simple setup would be
pclip aoijson --start "2017-06-01" --end "2017-12-31" --cloud "0.15" --inputfile "GJSON" --geo "C:\planet\myarea.geojson" --loc "C:\planet"
the output is always named as aoi.json.
Activate or Check Asset
The activate tool allows the users to either check or activate planet assets. This tool makes use of an existing json file sturctured for use within Planet API or the aoi.json file created earlier. This is a necessary step since the clip API can only work with those ID(s) which have been activated. In the future the list ID tool will check for number of activated id and wait for all of them to be activated before generating an ID list.
usage: pclip activate [-h] [--aoi AOI] [--action ACTION] [--asst ASST]

optional arguments:
-h, --help show this help message and exit
--aoi AOI Choose aoi.json file created earlier
--action ACTION choose between check/activate
--asset ASST Choose between planet asset types (PSOrthoTile
analytic/PSOrthoTile analytic_dn/PSOrthoTile
visual/PSScene4Band analytic/PSScene4Band
analytic_dn/PSScene3Band analytic/PSScene3Band
analytic_dn/PSScene3Band visual/REOrthoTile
analytic/REOrthoTile visual

An example setup for asset activation is the following
pclip activate --aoi "C:\planet\aoi.json" --action "activate" --asset "PSOrthoTile analytic"
List IDs
The next step is to list ID(s) that you have activated, this creates a temporary file containing the list of ID(s) which can be used to iteratively call the clips API. This is a modification of the activation function to use only the item id instead of item type and asset id and write to file for future use.
usage: pclip idlist [-h] [--aoi AOI] [--asset ASSET]

optional arguments:
-h, --help show this help message and exit
--aoi AOI Input path to the structured json file from which we will
generate the clips
--asset ASSET Choose from asset type for example:"PSOrthoTile
analytic"|"REOrthoTile analytic"

The example setup for this command is the following
pclip idlist --aoi “C:\planet\aoi.json” --asset “PSOrthoTile analytic”
Clipping with GeoJSON
A geejson file can be used directly to clip and query the area of interest and then submit clip process. I added this is a functionality but want to make clear that this does not take into consideration any other filters such as cloud cover or start and end date, and hence should be used only when you do not need to apply any filter.
usage: pclip geojsonc [-h] [--path PATH] [--item ITEM] [--asset ASSET]

optional arguments:
-h, --help show this help message and exit
--path PATH Path to the geojson file including filename (Example:
C:\users ile.geojson)
--item ITEM Choose from item type for example:"PSOrthoTile","REOrthoTile"
--asset ASSET Choose from asset type for example: "visual","analytic"

A simple setup for the JSON tool is the following
pclip geojsonc --path “C:\planet\aoi.geojson” --item “PSOrthoTile” --asset “analytic"
Clipping with JSON
This is the preferred style of submitting the clip requests using the IDlist we generated earlier. This is already structured before even activating assets and includes the additional filters you might have used for selecting the images.
usage: pclip jsonc [-h] [--path PATH] [--item ITEM] [--asset ASSET]

optional arguments:
-h, --help show this help message and exit
--path PATH Path to the json file including filename (Example: C:\users
ile.json)
--item ITEM Choose from item type for example:"PSOrthoTile","REOrthoTile"
--asset ASSET Choose from asset type for example: "visual","analytic"

A simple setup for the JSON tool is the following
pclip jsonc --path “C:\planet\aoi.json” --item “PSOrthoTile” --asset “analytic"
Downloading Clipped Imagery
The last step includes providing a location where the clipped imagery can be downloaded. This includes the zip files that are generated from the earlier step and include a download token that expires over time. This batch downloads the clipped zip files to destination directory
usage: pclip downloadclips [-h] [--dir DIR]

optional arguments:
-h, --help show this help message and exit
--dir DIR Output directory to save the assets. All files are zipped and
include metadata

A simple setup includes just the location to the download directory for the zipped & clipped files to be downloaded
pclip downloadclips --dir “C:\planet\zipped"

Sorting
As an additional measure and because it makes arranging and handling datasets easily, this setup comes completed with a sort tool. If a output directory is provided for the unzipped files, the tool unzips all files, moves the images and metadata to seperate directories and then deletes the original zipped files to save space.
usage: pclip sort [-h] [--zipped ZIPPED] [--unzipped UNZIPPED]

optional arguments:
-h, --help show this help message and exit
--zipped ZIPPED Folder containing downloaded clipped files which are
zipped
--unzipped UNZIPPED Folder where you want your files to be unzipped and
sorted

A simple would be the following (Images and metadata are sorted into an image and metadata folder inside the unzipped files folder)
pclip sort --zipped “C:\planet\zipped” --unzipped “C:\planet\unzipped”

Changelog
v0.2.6

Handles issues with urllib compatability in py2 and py3

v0.2.2

Improved Planet Key Handler
Added new tool to insepect planet account quota

v0.2.1

Thanks to commit suggested by Rabscuttler
Fixed issues with help text and installer

v0.2.0

Fixed issues with config files

v0.1.9

Now handles running and succeeded status better
Now enumerates during clip and download to allow user estimates on number of assets clipped and/or downloaded

v0.1.8

Includes required packages list within installer
Robust GEOJSON Parsing

v0.1.7

Fixed issues with processing visual asset types
The Clip function now handles error codes if the post response code is not 202(accepted for processing) then the error code and item and asset type is printed.

v0.1.6

Handles single time input API Key, this is needed only once to start the program
Fixed issue with base metadata folder during sort
Updated asset argument for asset activation to match styles

v0.1.5

Updated Requirements.txt to include pyshp
Fixed subprocess shell error, for now shell=True

v0.1.4

General Improvements

v0.1.3

General Improvements

v0.1.2

Tested on Ubuntu 16.04 and now handles permissions problem
Temporary files now written to config folders to avoid admin permission

v0.1.1

General Improvements