First of all, congrats on the V1 release of Temporal.
We were using only 1 worker for a couple of running workflows ( 50 000 ) approximately. Workflows instances are mostly waiting for signals to continue. We also have a poller Workflow ( only one instance ) with the NewContinueAsNewError every 2 minutes.
All seems to be working as expected, but the tasks stopped being processed by the worker.
Looking to metrics, Memory usage of the worker usually climbs all the way to 100% and drops down a few times. But one time it stopped executing tasks and the memory stayed on 100% until restart. After restart all tasks execution was normal again. The memory slowly climbed all the way up again, so we started another instance of the worker ( date : 10/02 ), but the result was the same, slowly climbed all the way up again . Bellow the graph of memory usage :
memory for each worker instance is 1024MB
I believe that workers keeps caches from workflows so that the bootstrap is faster when it needs to handle the same workflow afterwards. So the memory will always climb all the way to the top if the worker is not restarted?
If my assumption above is correct, is there any way to limit the memory available to be used by workers? Or maybe to change the way on how the “recycle” of workers cache is handled? Just to avoid reaching the 100% ?
Thanks for all the support,