"History scavenger does nothing on XDC standby cluster — how is secondary DB size supposed to stay bounded?"

Community Support
21

Hey @jiechenz, good timing, we’re mid-cleanup right now. Here’s what we’ve found across two of our XDC secondary clusters:

Staging hub-beta (fully cleaned, all 512 shards):

  • Deleted 19.2M orphaned history_tree rows and 250M history_node rows
  • Before cleanup: 71M tree rows / 3.1B node rows total
  • Pods stabilized after cleanup, history_scheduled_tasks backlog drained

Development hub-beta (in progress, 209/512 shards complete):

  • Deleted 86M orphaned history_tree rows and 2.36B history_node rows so far
  • Before cleanup: ~8.5B history_node rows, ~296M history_tree rows (sampled from shard 100)
  • Still running, heaviest shards hit 25-30M nodes each, taking 60-90 minutes per shard

Methodology: We wrote a script that for each shard builds a temp table of valid run_ids from executions, left-joins against history_tree to find orphans, then deletes matching rows from both history_node and history_tree. Batching at 200 trees per transaction with a 2s sleep between batches to avoid overwhelming Aurora during concurrent XDC replication load.

XDC backlog observation: We can see the replication queue depth via SELECT count(*) FROM replication_tasks on the primary (hub-alpha), this table holds tasks hub-beta hasn’t ACK’d yet. Dev-hub-alpha currently has ~74M queued tasks. History pod logs are dominated by VerifyVersionedTransition errors (mutable state not up to date), suggesting ordering dependencies are causing significant retry churn. The growth rate has dropped from ~8k/min to ~5k/min as we’ve cleaned up Aurora IO pressure, but it hasn’t started draining yet.

Curious whether the VerifyVersionedTransition retry behavior is expected during catch-up from a large backlog, or if there’s something we should tune.

22

We’ve been running a manual cleanup script against our XDC standby clusters and noticed something: the scavenger appears to handle stale branch tokens for workflows that still have a matching execution (e.g. ContinueAsNew chains), but does not clean up history_tree/history_node rows where the tree_id has no corresponding run_id in the executions table at all.

We accumulated billions of rows in this second category on both our staging and dev standby clusters, eventually triggering a Postgres XID wraparound emergency after running a large batch cleanup.

Is the scavenger expected to handle both classes of orphan? If not, is there a supported mechanism to clean up history_tree rows with no matching execution?

23

Hi, glad to hear staging is fully cleaned with pods stable.

On the scavenger handling orphan tree rows: it actually is designed to handle branches whose workflow has no matching execution row — yes, both classes of orphan. With historyScannerDataMinAge: 1h in place, the scavenger should be cleaning them up. Could you keep an eye on the skipped / failed metrics and flag if anything stands out?

On the XID wraparound — sorry to hear that. I think throttle the cleanup batches should help give room to catch up between rounds, I believe you have already done that.

On VerifyVersionedTransition retries — yes, expected behavior. Each task arrives tagged with the source’s expected mutable-state version; when the receiver is behind, it returns that SyncState error, and the processor automatically pulls the missing state delta and re-executes. 74M queued tasks is a lot though.

Hope the numbers get better in the next round.