asf-metadata 0.2.0

Description:

asfmetadata 0.2.0

asf_metadata
This repo contains all the Python 3.8 scripts related to manage ASF metadata. These are currently mostly dealing with ISO metadata.
Table of Contents

ISO template
Excel spreadsheet
ISO metdata functions
ISO metadata tools


ISO template
ISO metadata templates can be best maintained in a more user-friendly environment. I have settled for an Excel spreadsheet to achieve that.

The template mechanism has three different flows. All use an Excel spreadsheet to define the structure of the ISO metadata.
ISO metadata template generation
The ISO metadata's structure is stored in an Excel spreadsheet and contains identifiers for each metadata parameter that is variable. The structure and variables form the template that we can store in two formats: XML and JSON.
Format conversion
In order to convert the metadata between various formats, we need to know the structure of it. That is saved within an Excel spreadsheet.
This functionality is something I anticipate using for the NISAR mission, for which we only get metadata in ISO XML format. Using the structure from the spreadsheet allows to painlessly deal with the namespaces within the XML files.
ISO metadata generation
For the generation of ISO metadata, we need the structure from the Excel spreadsheet. The template parameters need to be exchanged by their actual values. This will be achieved by extracting the necessary information based on the product files, replacing the functionality that existed in the old MapReady framework.

Excel spreadsheet
The Excel spreadsheet has a number of worksheets.
ISO metadata structure

This worksheet provides the relevant parts of the ISO metadata, most importantly its structure and the namespaces used within that structure.
Namespaces
One of the biggest nuisances about ISO XML metadata is the namespaces.

This worksheet links each parameter to its respective namespace.

This worksheet defines the URLs for the individual namespaces needed in the XML declarations.

ISO metadata functions
I developed a number of functions that perform the various tasks outline above.
Extracting information from templates
The template information in the Excel spreadsheet is extracted (using the isoTemplate2lists function) into a number of lists: the flattened template structure, the individual metadata parameters, and their respective values.
Conversion of metadata formats
In order to convert an XML file to JSON format, we need to extract the individual metadata parameters out of the ISO metadata XML file using the get_metadata_values function.
Replacing placeholders with actual values
The templates stored in the Excel spreadsheets contain ISO parameter placeholders. The product_dictionary2values function replaces these placeholders, saved in a product dictionary, with the actual metadata values.
Generating ISO metadata from product files
ISO metadata can be generated by extracting the necessary information based on the product files (using the generate_product_dictionary function). Depending on the product type, specific conversion functions are called:

RTC products: gamma_rtc_metadata function

Building metadata structures
The ISO metadata is saved in two different formats: XML and JSON. The XML is using a tree structure that is built with the iso_xml_structure function, while the JSON structure is formed by a complex nested dictionary using the iso_dictionary_structure function.
Writing metadata to file
There are two separate functions to save the metadata to a file format. meta_xml_file is the function that generates an ISO XML metadata file, while meta_json_file saves the metadata in JSON format.
Cleaning up metadata
There are two separate functions to clean up the metadata structures. clean_xml_structure (for XML files) and clean_json_structure (for JSON files) is the function that removes metadata for product files (digital elevation model, incidence angle map, scattering area map) that are only optionally included.

ISO metadata tools
I developed a number of tools that perform the various tasks outline above.
Generate ISO template file from Excel spreadsheet
usage: generate_iso_template.py [-h] [-dem DEM] excelFile isoBase
Generate ISO template file from Excel spreadsheet
positional arguments:
excelFile name of the Excel template spreadsheet
isoBase basename of the ISO metadata template file
optional arguments:
-h, --help show this help message and exit
-dem DEM name of DEM template spreadsheet

Generate ISO metadata file from Excel spreadsheet
usage: generate_iso_metadata.py [-h] [-product PRODUCT] [-data DATA] [-dem DEM] meta_file log_file iso_base
Generate ISO metadata file from Excel spreadsheet
positional arguments:
meta_file file path of product metadata such as a "manifest.safe" or leader file
log_file name of RTC processing log file
iso_base basename of the ISO XML metadata file
optional arguments:
-h, --help show this help message and exit
-product PRODUCT name of the product type (default: gamma_rtc)
choices: ["gamma_rtc"]
-data DATA name of the data source (default: sentinel)
choices: ["sentinel"]
-dem DEM DEM type used for terrain correction
choices: ["copernicus"]

At the moment, I have only added one function gamma_rtc_log2meta that analyzes GAMMA RTC log files for metadata. The corresponding product type is 'gamma_rtc'.
Author
Rudi Gens