have an xdc setup and have about 1500 worflows with reasonable amount of history (say 200 to 300 activity)… i am seeing that the first time replication is actually takeing some time ( 3+ hours already), i tried increasing the number of front ends in my active cluster and no of workers in the standby…

Thanks for all the help @yux Finally, we figured out that ther was no issue with the configurations. It appears that most of my workflows keeps sleeping/waiting for an event. and By design temporal do not replicate sleeping workflows… So one way of force replicating is to send a fake signal to …

[image] madhu: any metrics i can track to figure out if active and standby seems this is answered here XDC Replication in Practice - #25 by yux

for instance my active cluster i have 752336 entries in replication_tasks and 140 entry in replicaiton_tasks_dlq table. in my secondary (standby) cluster i have 0 task in replication_tasks and 6158 entries in replicaiton_tasks_dlq I dont see these numbers chaning for a very long time now (5+ ho…

[image] madhu: Between which component is responsible for replication? worker or hisotry? The history does the replication job. Besides adding hosts, there are some replication configs that you can speed up the replication rate. history.ReplicationTaskFetcherAggregationInterval (in remote/…

[image] yux: The history does the replication job. Besides adding hosts, there are some replication configs that you can speed up the replication rate. shoudl this be done in dynamic-config.yaml?

[image] yux: One issue I see with your information, when the replication tasks go into DLQ, those tasks encounter errors and cannot be replicated. Do you see any error related to replication from the remote/standby cluster to help us debug the issue? In my active worker i see errors like {"l…

[image] madhu: "Failed to get replication tasks" This error indicates the active cluster did not connect to the standby. Try to remove the standby cluster and add it again to see if you still see this error in active cluster. tctl admin cluster remove-remote-cluster --cluster standby tctl a…

[image] yux: here anything caused pod restart? resource exhausted? no. resource usage is not even 30%, cpu or memory… network wise i dont see packet losses. I tried doing a rrc in both primary and standy and it did not help,i infact tried deleting the cluster_metadata_info table as well. tha…

[image] yux: This error indicates the active cluster did not connect to the standby. Try to remove the standby cluster and add it again to see if you still see this error in active cluster. i think both the clusters are able t connect to each other , when i bring down my standy by making repl…

We released a patch to some issues around multi-clusters last Friday. Could you also try this patch to see if it resolve the issue?

thanks @yux , i upgraded to 1.14.4 ,unfortunately the problem still persists.

Whats optimal settings for quick xdc replication? what metrics can one track?

Community Support

madhu January 21, 2022, 5:20am 2

seems this is answered here XDC Replication in Practice - #25 by yux

Topic		Replies	Views
XDC Replication in Practice Community Support java-sdk , xdc	26	3730	September 18, 2023
Errors in frontend and hisotry Community Support history , worker , grpc	26	2464	April 9, 2022
Failed to retrieve replication message and Failed to get replication tasks from client errors Community Support replication	6	683	July 8, 2024
XDC Replication discrepancies Server Deployment replication	0	181	February 22, 2024
XDC Namespace create/update events are not getting replicated/applied Community Support history , worker , xdc , server	6	619	August 30, 2022

Whats optimal settings for quick xdc replication? what metrics can one track?

Related topics