Cross DC architecture

Hi all, we would like to have a cross dc function for temporal service, so that we would like to know the relationship between Kafka and Elasticsearch, how these two can help with cross dc replication? Thanks.

1 Like

Current implementation (v1.3.2):

  1. XDC is using Kafka for replication log distribution, e.g. source cluster sends replication log to Kafka, target cluster polls from Kafka for replication logs.
    NOTE: We are in the progress of removing Kafka for XDC reducing number of dependencies.

  2. Advanced visibility (backed by ES) is using Kafka for records buffer and batch processing.
    NOTE: We are in the progress of removing Kafka for Advanced visibility reducing number of dependencies.

few more threads on this

I noticed that in the docs there is a reference to an rpc replication consumer type. Is this the path forward to remove the dependency on kafka? If so is this feature already working and just not documented completely yet, or is it still a work in progress?

yes.

the cross DC feature is still experimental, but you can run it (using RPC stack)

https://github.com/temporalio/temporal/blob/master/config/development_active.yaml
https://github.com/temporalio/temporal/blob/master/config/development_standby.yaml