Is there a way to see what is on the task queues?

Community Support
,
1

Hi,

I was wondering if there is a way to visualize in the UI what’s on the task queue to see what haven’t been processed?

Thanks,
Derek

2 Likes
2

So we do not directly support this today from the UI or even the API. That being said, depending on your use case there may be some capabilities that solve your problem. Can you give me a bit more info on your use case?

5

I think it would be useful for

  • debugging: It’s nice to be able to see where the information goes in reference to the task queue that user set in the workflow/activity code. At least for me, I find it useful in seeing if the events are being pushed in the correct queue and just to understand what’s going on
  • It’s also useful to see what the backlog of events on the task queue are. Imagine if the your system goes down and you want to see what events are left to process. Also knowing how much events are left gives you an idea of how long it will take to recover from the backlog of queued items.

Thanks,
Derek

1 Like
6

Are there metrics that give some information about the task backlogs?

1 Like
7

You can use schedule to start latency metric. Without backlog it is close to 0 and keeps increasing with backlog size. We do have plans to add counts and possibly list operation for a task list.

2 Likes
8

Maxim, what metric are you referring to for schedule to start latency? I see the schedule_to_start_timeout counter, but I don’t see a timer metric for schedule to start.

9

It’s a client-side metric, see Go sample here. We are working on a java sample for it atm as well.
For Java, this thread that has a lot of info on how to set it up before the sample is committed.

10

@tihomir @maxim ,

We have got similar requirement where we would want to know the count of pending messages/events from the task queue. Is there any metric/api in the latest server version, we are on 1.20.1 server version

11

Is there any update on this? We want to use the number of messages in queue for scaling purpose. We already have it on schedule to start latency metrics but that doesn’t give a clear picture on how many messages are in queue that we would be beneficial for us.

12

Don’t think this is available as of yet.

one way to measure backlog currently via service metrics is service_latency histogram metric for operations PollActivityTaskQueue and PollWorkflowTaskQueue, sample query:

histogram_quantile(0.95, sum(rate(service_latency_bucket{operation=~"PollActivityTaskQueue|PollWorkflowTaskQueue"}[5m])) by (operation, le))

On the SDK metrics side, as previously mentioned you would look at latencies of
workflow_task_schedule_to_start_latency and activity_schedule_to_start_latency