Temporal Server Memory Bloat: Anyone Found a Fix?

Community Support
,
1

We’re seeing multi-GB RAM in both Temporal Server (History service) and Postgres, even though we:

  • Run only a handful of workflows per user; each runs every minute and continue-as-new to keep histories short.
  • Slashed cache sizes and retention:
history.cacheInitialSize: 8 # default 128 
history.cacheMaxSize: 16 # default 512 
history.eventsCacheInitialSize: 8 # default 128 
history.eventsCacheMaxSize: 16 # default 512 
dataRetention: 1h # was 1D

What we observe

  1. RAM rises until it hits the retention window, then plateaus—but still far too high.
  2. LRU cache keeps completed/continued-as-new workflow histories in memory until another running workflow evicts them.
  3. Community threads call this a “memory leak”; wasted RAM looks like 50-95 % of total.

Has anyone found a workaround or settings that truly flush unused cache or reduce memory usage? All insight welcome—thanks!

2

Also posted in Slack here

Its possible to set the data retention policy to 1hr even though 1day seems to be the docs said minimum. We are using the https://hub.docker.com/r/temporalio/auto-setup for now. Anytime we reset the DB and re-deploy the memory drops like a rock and then gradually increases every minute linearly till the data retention period is hit. We really would only like to use memory for running workflows and are hoping there is some configuration or setting we can set to achieve this. For clarity the first few times the workflows run for all users after a db reset and redeploy the memory is nice and low, each minute it increments by the number of completed/continued-as-new workflows.