We have a requirement to scale workers basis the task-queue size, how do i retrieve the task-queue size metric (using this metric , from outside , i am planning to scale the worker pods in eks using horizontal pod scaling), plz help
Is it possible to query the database directly to retrieve the queue size, if so , plz share the query as well.
Don’t think there is a way to get exact task queue size but you can look at backlogs:
Server metrics, service_latency_bucket, for example:
histogram_quantile(0.95, sum(rate(service_latency_bucket{operation=~"PollActivityTaskQueue|PollWorkflowTaskQueue"}[5m])) by (operation, le))
(no workers polling on tq can cause backlog, which increases these latencies)
From SDK metrics side watch increased latencies of workflow and activity task schedule to start latencies: workflow_task_schedule_to_start_latency, activity_schedule_to_start_latency