What are the best metrics to autoscale each cluster service on?

[RichRM]

Is there a list anywhere that shows which metrics we should be watching/using to scale each individual service?

TIA!

[tihomir]

Hi, don’t think we have such docs yet, would start with metrics mentioned in post here. For service errors and latencies you can change the service_name if needed.

[RichRM]

What about these, are they also worth scaling the individual cluster services on?

Suggested metrics to autoscale Temporal workers on

what are the general metrics you recommend autoscaling workers on? Would use a combination of SDK and server metrics. SDK: Starting point can be the worker tuning guide in doc and metrics: worker_task_slots_available: Gauge metric, defines how many task slots are available for workers to process tasks. It should be > 0, otherwise workers would not be able to keep up processing tasks. Sample Prometheus query: avg_over_time(temporal_worker_task_slots_available{namespace="default",worker…

Sorry, I get confused between “Temporal workers” and the “worker” service.

[tihomir]

Yeah those are good as well.

I get confused between “Temporal workers” and the “worker” service.

Workers are worker processes that you deploy and that execute your code. Worker service is one of Temporal server services (frontend, history, matching, worker). It’s mainly responsible to run background system workflow on the temporal-system namespace.

[RichRM]

@tihomir

Workers are worker processes that you deploy and that execute your code.

Thanks, I think I am slowly getting it

Which cluster service do those “worker processes” run inside? Or does it depend a lot on the exact code?

[tihomir]

Your workers independent of the service (temporal server). You can deploy them any way you like. Workers poll service (specifically frontend service) for workflow and activity tasks.
Maybe see this video that might help with understanding.