We’re running Temporal 1.31.0 (via the helm chart) with XDC replication across two aurora postgres clusters (temporal-primary, temporal-secondary). We’ve noticed temporal-secondary consistently runs significantly larger than temporal-primary. On a single shard in our development environment temporal-secondary has 649K history_tree rows and 16M history_node rows vs temporal-primary’s 5K trees and 138K nodes, roughly 120x larger. This makes temporal-history nodes run fine with 8gb ram on primary, but OOM with 64gb ram on secondary.
My understanding is that temporal-primary’s scavenger deletes history_node/history_tree rows locally when retention expires, but are those deletes not propagated to temporal-secondary via XDC?
If that’s correct, is history.enableDeleteWorkflowExecutionReplication (defaults false) the right way to address this, and does it cover scavenger/retention-driven deletes or only explicit DeleteWorkflowExecution API calls?
I figured primary/secondary would be nearly identical in case you have to fail over to secondary some day, but maybe i’m not understanding XDC correctly