ApproximateBacklogCount and ApproximateBacklogAge prometheus stats?

Community Support Server Deployment
1

Are these taskqueue properties available via prometheus stats?

  • ApproximateBacklogCount
  • ApproximateBacklogAge
  • BacklogIncreaseRate

I would like to start using these to make scaling decisions

2

Was told there are plans for this in the future, currently you need to scrape this data via describe task queue api and then push to prom as custom metric.

1 Like
3

@tihomir

For clarity for me and perhaps other users of Temporal, could you share the command to run or share a snippet of java code to accomplish this?

Would be great to get an linke to a java equivalent of this code: Class: DescribeTaskQueueRequest | Temporal TypeScript SDK API Reference

4

share a snippet of java code to accomplish this?

something like this should work for you:

 DescribeTaskQueueResponse res =
        service
            .blockingStub()
            .describeTaskQueue(
                DescribeTaskQueueRequest.newBuilder()
                    .setNamespace("default")
                    .setTaskQueue(TaskQueue.newBuilder().setName(TASK_QUEUE).build())
                    .setApiMode(DescribeTaskQueueMode.DESCRIBE_TASK_QUEUE_MODE_ENHANCED)
                    //                    .setIncludeTaskQueueStatus(true)
                    .setReportStats(true)
                    //                    .setReportTaskReachability(true)
                    //                    .setReportPollers(true)
                    .build());

// approx backlog count for workflow tasks for this tq (we assume here task queue is unversioned), for activity tasks use types info 2, 
// see https://github.com/temporalio/api/blob/master/temporal/api/enums/v1/task_queue.proto#L53-L61

res.getVersionsInfoMap()
            .get("") // unversioned
            .getTypesInfoOrThrow(1) // workflow tasks 1
            .getStats()
            .getApproximateBacklogCount());
2 Likes
5

Do you know if there is a way to get this across all task queues? I only see how to describe individul task queues.

If not, is it possible to query this from the DB directly? Basically I would like to start scaling based on the “worst” queue. But in general I would like to plot all of the queues for visibility.

6

@tihomir which version of the sdk is .getStats() introduced in exactly?
We’re using 1.23.2 at the moment.
If I would bump the sdk, which version of the server is needed to get this to work?

Ok, answered my own question, it’s 1.25.0 Release v1.25.0 · temporalio/temporal · GitHub

Another question, @tihomir could you explain the following things to us please?

  • .get("") // unversioned what does the not versioned relate to exactly and how do I know when to change it?
  • getTypesInfoOrThrow(1) // workflow tasks 1 why 1, and what is stored in index 0 exactly?
  • when I query Temporal the approx backlog -count I see is roughly equal on all nodes, the approx-age however seems to be volatile and quite different per node, any idea how that can be? I have some ideas on how this might work but dont want to put words in your mouth.

Thanks in advance, regards, Frank