datastax/pulsar-mirror repository overview

⁠pulsar-mirror

Pulsar's mirroring feature maintains a replica of all messages of an existing Pulsar cluster. The tool mirrors messages from a source Pulsar cluster to a destination Pulsar cluster. The tool uses a Pulsar consumer to consume messages from the source cluster, and re-publishes those messages to the destination cluster using a producer.

⁠Feature list

• Replicate all topics under a single tenant, or a list of tenants (public tenant is excluded)
• Auto create namespaces and topics
• Namespace mapping (maps old source namespace to destination namespace name, this is optional feature)
• Blacklist topics
• Expose Prometheus metrics against each source and destination topics
• A basic Kubernetes deployment yaml support
• Stateless design (all statement are kept in either source or destination Pulsar cluster)
• Predefined load balancing of multiple mirroring processes
• Individual topic migration with topic fullname mapping
• Schema replication
• Auto-create partition topics
• Message transformation/fitering ?

⁠Design notes

1. The source message's message ID is saved as sourceMsgID under the message property in the destination Cluster's message.
2. This tool does not create any subscription on the destination cluster. Please configure sufficient retention to keep mirrored messages on the destination topics.
3. Configuration DisableSecondPass can stop any mirrored message to be sent again. This prevents circular message loop in the system.

⁠Dev

⁠Run from the source

cd src
go run main.go

⁠Docker build

make container

⁠Kubernetes

We suggest to install pulsar-mirror as a deployment pod within the same pulsar namespace

To install

kubectl apply -f ./kubernetes/deploy-mirror.yaml

To delete

kubectl delete deploy pulsar-mirror-deployment