Hi,
As per https://temporalio.slack.com/archives/CTRCR8RBP/p1605352484031900
My temporal database is growing quite large. (1TB+)
I’ve set the retention tot 0 using tctl: tctl --ad temporal:7233 --ns mynamespace namespace update --retention 0
However, the database size is still growing and I don’t want to impact cron and new workflows…
Any help? Thanks!
Do you know the table which is taking most of the space?
Updating retention only applies to workflow executions which closes after the retention is updated. All workflow executions closed previously still need to wait on the timer based on previous retention policy for them to be deleted.
Another cause could be that we create a 10 year timer for each workflow timeout and potentially activity execution. Although these timers are tiny but they can add up if you are starting workflows/activities at a very high rate. We have a PR out to address this issue. Should be included in our next release.
Seems to be temporal, I guess it resized but won’t size back down:
temporal | 111 GB
temporal_visibility | 17 GB
Can you provide further breakdown within the keyspace? What was your retention before you set it to 0? If the size came down then most probably retention timer for your old workflow executions have started to kick in. This would result in deleting all resources consumed by previous workflow executions.
Also can you make sure to set a reasonable timeout for workflow execution yourself until PR to address very long timer is merged and released.