pdp-kafka-reader 0.0.5
Last updated:
0 purchases
Description:
pdpkafkareader 0.0.5
PDP Kafka Reader
Requirements
git
python 3.6+
pip
Also, you need access to the git repository. Generate and use ssh keys in Skyway Bitbucket.
Install
pip install pdp_kafka_reader
Usage
CLI
You can use kafka-reader CLI tool to extract data into from a specific topic. An example of usage:
kafka-reader export-avro -k kafka-options.json -s schema.json -t my_kafka_topic -o out.parquet
Check all options with kafka-reader -h.
Python KafkaReader
import json
from pdp_kafka_reader.kafka_reader import KafkaAvroReader
kafka_options = {
"kafka.bootstrap.servers": "my-kafka-server:9092",
"subscribe": "test_avro"
}
avro_schema = open("schema.json").read()
reader = KafkaAvroReader(spark)
df = reader.read_avro(kafka_options, avro_schema, "my_kafka_topic")
df.show()
Testing
Testing environment in defined in docker-compose.yml. Start docker containers and run tox.
License:
For personal and professional use. You cannot resell or redistribute these repositories in their original state.
Customer Reviews
There are no reviews.
