Workloads & Metrics

Community Support
, , ,
1

Hi everyone,

I would like some advice on the correct management of the options of the temporal workers.
I have a pre production environment that needs to launch 100 (more or less) activities in parallel, but initially I was getting java heap space problem.
After setting parameters such as ActivityPollThreadCount = 5, MaxConcurrentActivityExecutionSize = 20, MaxTaskQueueActivitiesPerSecond = 0.2 and MaxWorkerActivitiesForSecond = 0.2, now I rarely get heap space, but it’s not completely gone.
Can I check, with some specific metrics, if the ram memory or the cpu of a microservice is usable or not before launching a scheduled activity? Otherwise, do you have any advice on this to avoid starting too many activities at the same time, based on memory usage? (each activity, based on its input, can use more or less memory based on the set of data to be processed)

Thx a lot
A.

3

You could implement a workflow that would manage the worker. This workflow would receive signals with requests to execute the activity and signals from the worker to indicate that there is available capacity. Then based on this information, it would be able to execute activities on that worker according to whatever policy is appropriate.

5

Thx for response Maxim.
Then there isn’t any native implementation to manage it? Or i could use some grafana’s metrica to achieve it?
Last question, in ur opinion the best way to manage it is create this mechanism that u explained or use kube metrics on pod and scale it when theshold was reached?

I appreciate ur work and tips.
Thx a lot.
A.

6

Filed Add ability to dynamically limit number of parallel activities · Issue #1042 · temporalio/sdk-java · GitHub

1 Like