Should the secondary XDC cluster be far larger than the primary?

Community Support
,
1

We’re running Temporal 1.31.0 (via the helm chart) with XDC replication across two aurora postgres clusters (temporal-primary, temporal-secondary). We’ve noticed temporal-secondary consistently runs significantly larger than temporal-primary. On a single shard in our development environment temporal-secondary has 649K history_tree rows and 16M history_node rows vs temporal-primary’s 5K trees and 138K nodes, roughly 120x larger. This makes temporal-history nodes run fine with 8gb ram on primary, but OOM with 64gb ram on secondary.

My understanding is that temporal-primary’s scavenger deletes history_node/history_tree rows locally when retention expires, but are those deletes not propagated to temporal-secondary via XDC?

If that’s correct, is history.enableDeleteWorkflowExecutionReplication (defaults false) the right way to address this, and does it cover scavenger/retention-driven deletes or only explicit DeleteWorkflowExecution API calls?

I figured primary/secondary would be nearly identical in case you have to fail over to secondary some day, but maybe i’m not understanding XDC correctly

2

is history.enableDeleteWorkflowExecutionReplication (defaults false) the right way to address this,

its not. namespace-retention based deletions explicitly bypass replication, meaning both clusters active and passive do their own namespace-based retention clenup

this dynamic config however is pretty useful when you do explicit deletions on active, for example run DeleteWorkflowExecution or batch delete. in those cases the delete tasks would be replicated (if flag is set to true)

looks like some issue on passive with retention deletion…
archival is in critical path, would check archival issues on passive
maybe timer queue lag on secondary
worth checking as it sounds something is going on on secondary that needs to be looked at