Worker Memory usage

Community Support
,
1

Hello,

First of all, congrats on the V1 release of Temporal. :slight_smile:

We were using only 1 worker for a couple of running workflows ( 50 000 ) approximately. Workflows instances are mostly waiting for signals to continue. We also have a poller Workflow ( only one instance ) with the NewContinueAsNewError every 2 minutes.

All seems to be working as expected, but the tasks stopped being processed by the worker.

Looking to metrics, Memory usage of the worker usually climbs all the way to 100% and drops down a few times. But one time it stopped executing tasks and the memory stayed on 100% until restart. After restart all tasks execution was normal again. The memory slowly climbed all the way up again, so we started another instance of the worker ( date : 10/02 ), but the result was the same, slowly climbed all the way up again . Bellow the graph of memory usage :

image
image2798Ă—440 96.6 KB

memory for each worker instance is 1024MB

I believe that workers keeps caches from workflows so that the bootstrap is faster when it needs to handle the same workflow afterwards. So the memory will always climb all the way to the top if the worker is not restarted?

If my assumption above is correct, is there any way to limit the memory available to be used by workers? Or maybe to change the way on how the “recycle” of workers cache is handled? Just to avoid reaching the 100% ?

Thanks for all the support,
Pedro Almeida

2

Try setting workflow.SetStickyWorkflowCacheSize.

1 Like
3

Hello Maxim,

SetStickyWorkflowCacheSize seems to be working.

image
image2820Ă—476 74.6 KB

Thanks for the help.

Pedro Almeida

1 Like
4

Hello Maxim,

i have same issue in java , where i can set workflow.SetStickyWorkflowCacheSize in java-sdk ?

Thanks,
Rz

5

@Rizdian_Dinata
for Java set it via WorkflowFactoryOptions#setWorkflowCacheSize, for example:

WorkerFactory factory = WorkerFactory.newInstance(client,
            WorkerFactoryOptions.newBuilder()
                    .setWorkflowCacheSize(myWorkflowCacheSize)
                    .build());

see also worker tuning page in docs.

1 Like
6

would like to ask regarding this function as well PurgeStickyWorkflowCache
is it recommended to set this during the worker of workflow termination in pod?

7

Hi @Meli_Lee,
for Go SDK:

worker.SetStickyWorkflowCacheSize(cacheSize) sets the cache size for sticky workflow cache. This cache is shared between all workers running in the same process and must be called before any workers are started.

worker.PurgeStickyWorkflowCache() resets this sticky workflow cache across all workers. It can/should be only called when all workers are stopped.

Hope this helps.

8

so if we don’t set it during service termination is it fine right ?

9

If I understand the question correctly:

Worker cache is in memory, and calling PurgeStickyWorkflowCache after worker service terminated is not needed cause it would have no effect.
Once your workers are back up the server will treat them as completely new workers.

1 Like
10

Hi @maxim,
I have an problem, when I send 500 request in the same time, It will consume a lot of memory, then I wait for a long time, but memory not decrease, please help me, thank you.

11

Can you give more info on this? Do you mean memory utilization on your worker processes?

12

Hi @tihomir

smarter-1w
smarter-1w1537Ă—268 19 KB

You can see, ram memory alway increase, not decrease.

13

HI @jack_king I have the same issue, how did you fix it?

14

I also encountered the memory leak problem when using Python SDK. How should I solve it?

15

Duplicate of We now have a memory leak problem - #2 by alchemya

16

Hello @maxim

We configured the values ​​of “max_cached_workflows” and “max_concurrent_activities” in our python code,but our memory is also increasing.
I searched all the posts about memory leaks in the forum and found this answer, but I don’t know how to configure the workflow.SetStickyWorkflowCacheSize method in the python sdk,Where can I set it?
Thanks,
alchemya

1 Like
17

We are also constantly facing this issue. Any update on this @maxim ?

18

I don’t think anything changed about this. We are not aware of any memory leaks in the SDKs. Setting the cache size is a way to limit a worker’s memory usage.

If you believe an SDK has a memory leak, we would need a reproduction to investigate.