Observing significant memory consumption increase after 1 hour of the workflow completion

We are doing some experiments around the Temporal Java SDK client memory consumption. We also developed a small tool to keep track of the application process’s memory consumption and plot it

These experiments are performed in such a way: at the beginning of measurement period, we trigger the workflow which will run exactly 1 hour. After workflow completes, we won’t trigger anything else and just keep the measurement ongoing to see the performance of temporal client in the idle state

To set up a baseline, the application that we embed our client takes around 710 MiB of memory. Its memory consumption is very stable and won’t vary too much over the time.

In several experiments we observe one common patterns

  1. There is a significant increase of memory consumption happens around 1 hour after the completion of workflow. This can be indicated by the sudden steep increase of memory in the plot

For example,


and many more

In the last example, the workflow completes at 2024-02-16 10:22:04.77

Looking at the memory consumption, we can see a sudden increase occurs after 1 hour at 2024-02-16 11:23:12

from 756.83MiB to 767.37MiB.

We are looking for some potential explanation to understand why is always a sudden increase of memory consumption on the SDK client side one hour after last workflow completion? Understanding this behavior will help us better tailor the Temporal client based on our requirements. Thanks

Hi @maxim and @tihomir ,
I am seeing the same behavior. Is there a memory leak due to periodic workflow? If not, is there a way we can keep memory usage stable if no workflow is running?

Please help!

I would recommend taking the heap dump of the process before and after 1 hour to see what objects consume the space.