This is a standalone application that allow you to mirror Avro, String & Bytes messages
291
This docker image allow you to mirror Avro, String/JSON or Byte data from a Source to Target Kafka Clusters.
IMPORTANT : Until this current version, only String KAFKA message key is accepted => other types will be soon available
| Config | Description | Default |
|---|---|---|
| SOURCE_BOOTSTRAP_SERVER | Bootstrap server of source Kafka cluster | localhost:9092 |
| SOURCE_GROUP_ID | The group ID to be used by the consumer on source Kafka cluster | avro-mirror-group-id |
| SOURCE_SCHEMA_REGISTRY_URL | The Schema Registry URL of the source Kafka cluster | http://localhost:8081 |
| SOURCE_WHITELIST_TOPICS | The comma separated list of the topics to be mirrored from the source cluster. Data types on source topics could be : AVRO, String (simple text or JSON) or Bytes. Before 1.0.0 version, only AVRO is allowed | input_topic_1,input_topic_2 |
| TARGET_BOOTSTRAP_SERVER | Bootstrap server of the target Kafka cluster | localhost:9092 |
| TARGET_SCHEMA_REGISTRY_URL | The Schema Registry URL of the target Kafka cluster | http://localhost:8081 |
| TARGET_TOPIC_PREFIX | The default prefix to be used on the target cluster. Ex: If the source topic is input_topic and the prefix is west1- the target topic will be west1-input_topic | |
| IGNORE_PRODUCING_ERROR | Log and ignore errors on producing messages to target cluster | true |
To override any configuration on the consumer of the Source Kafka Cluster, you have just to add a new Env variable prefixed by SOURCE_CONSUMER_.
Ex:
If you would like to override the auto.offset.reset to earliest => Your Env variable will look like that :
SOURCE_CONSUMER_AUTO_OFFSET_RESET=earliest
To override any configuration on the producer of the Target Kafka Cluster, you have just to add a new Env variable prefixed by TARGET_PRODUCER_.
Ex:
If you would like to override the compression.type to gzip => Your Env variable will look like that :
TARGET_PRODUCER_COMPRESSION_TYPE=gzip
docker run -d -e SOURCE_BOOTSTRAP_SERVER=source-cluster:9092 \
-e SOURCE_GROUP_ID=my-group-id \
-e SOURCE_SCHEMA_REGISTRY_URL=http://schema-registry:8081 \
-e SOURCE_WHITELIST_TOPICS=avro-topic1,string-topic2,bytes-topic3 \
-e TARGET_BOOTSTRAP_SERVER=target-cluster:9092 \
-e TARGET_SCHEMA_REGISTRY_URL=http://target-cluster-schema-registry:8081 \
-e TARGET_TOPIC_PREFIX=mirror_ \
-e IGNORE_PRODUCING_ERROR=false \
--name standalone-mirror kafkaetech/standalone-avro-mirror:1.0.0
Content type
Image
Digest
sha256:1b02bc9e5…
Size
223.5 MB
Last updated
over 3 years ago
docker pull kafkaetech/standalone-mirror:1.2.0