GitLocker: The Coding Marketplace

Description:

monascatransform 2.0.0

Team and repository tags

Monasca Transform

Use Cases handled by Monasca Transform
Operation
Architecture
To set up the development environment
Generic aggregation components
Create a new aggregation pipeline example
Original proposal and blueprint

Monasca Transform
monasca-transform is a data driven aggregation engine which collects,
groups and aggregates existing individual Monasca metrics according to
business requirements and publishes new transformed (derived) metrics to
the Monasca Kafka queue.

Since the new transformed metrics are published as any other metric
in Monasca, alarms can be set and triggered on the transformed
metric.
Monasca Transform uses Apache Spark to aggregate data. Apache
Spark is a highly scalable, fast, in-memory, fault tolerant and
parallel data processing framework. All monasca-transform components
are implemented in Python and use Spark’s PySpark Python API to
interact with Spark.
Monasca Transform does transformation and aggregation of incoming
metrics in two phases.

In the first phase spark streaming application is set to retrieve
in data from kafka at a configurable stream interval (default
stream_inteval is 10 minutes) and write the data aggregated for
stream interval to pre_hourly_metrics topic in kafka.
In the second phase, which is kicked off every hour, all metrics
in metrics_pre_hourly topic in Kafka are aggregated again, this
time over a larger interval of an hour. These hourly aggregated
metrics published to metrics topic in kafka.

Use Cases handled by Monasca Transform
Please refer to Problem Description section on the
Monasca/Transform wiki

Operation
Please refer to How Monasca Transform Operates section on the
Monasca/Transform wiki

Architecture
Please refer to Architecture and Logical processing data flow
sections on the Monasca/Transform wiki

To set up the development environment
The monasca-transform uses DevStack as a common dev environment. See
the README.md in the devstack directory for details on how to include
monasca-transform in a DevStack deployment.

Generic aggregation components
Monasca Transform uses a set of generic aggregation components which can
be assembled in to an aggregation pipeline.
Please refer to the
generic-aggregation-components
document for information on list of generic aggregation components
available.

Create a new aggregation pipeline example
Generic aggregation components make it easy to build new aggregation
pipelines for different Monasca metrics.
This create a new aggregation pipeline example shows how to create
pre_transform_specs and transform_specs to create an aggregation
pipeline for a new set of Monasca metrics, while leveraging existing set
of generic aggregation components.

Original proposal and blueprint
Original proposal: Monasca/Transform-proposal
Blueprint: monasca-transform blueprint