Hi Colleagues! Could please suggest how workflow history clean up works. In case if Archival feature on domain is disabled and retention is > then 0 days
Temporal has an implementation of queue running within each history service shard which allows querying tasks based on visibility timestamp. This is a very key primitive and used for any use case within the system where durable timers are needed, including enforcing retention.
When user registers a domain they need to provide value for retentionInDays. This value is used to create DELETE_HISTORY_EVENT timer task for the future on the last state transition to close(complete, failed, timedout, canceled, terminated) a workflow execution. This task is dispatched to history shard after a delay specified by retention period. The processor for DELETE_HISTORY_EVENT task then issues delete query to the DB to remove any resources associated with this execution.
In the situation where archival is turned on, the same timer queue processor for DELETE_HISTORY_EVENT sends a signal to internal system workflow which is responsible for first archiving the workflow execution before deleting it from the database.
2 Likes
Hey @samar can you explain what the role of the scavenger that is run on the worker? It also seems delete history. Also, why is the scavenger only scheduled when using Cassandra as persistence and not MySQL?
We run few different kind of scavengers to automatically detect and correct consistency problems caused by DB issues by doing full scans. We have one scavenger running which scans all the executions in background and detect some cassandra specific problems. For instance one problem it looks out for is caused by resurrection of deleted records due to cassandra tombstone resurrection. This is the reason this scavenger only runs for cassandra persistence as this issue does not exist in MySQL persistence.
As far as deletion of history events, these scavengers due not play any role in that. That is solely done through Workflow Deletion timer task running in history service or in the case of archival a signal is sent to archival system workflow which deletes the history after workflow is successfully archived.
1 Like