Hi, We are currently doing stress testing on our application which uses Temporal for micro-service orchestration. Our current setup is a GKE cluster as defined below - 2 T2D Nodes with 32 cores, 64 GB memory having 8 application pods - with each pod having 50 workflow pollers, 50 acitivity polle…

From the provided info it looks as latencies are happening due to your workers being under-provisioned. Also it looks from description that you are shooting for > 100K state transitions per second as far as performance goes, is this true?

Each workflow execution takes 250 ms, there are 8 worker pods with 50 pollers each for workflow and activity, concurrent execution size of 200(we even tried with 300), and we are hitting 250 Workflows per second with 3 activities each activity taking 40-45 ms as can be seen from the trace attached. …

Some of the things we tried since we posted the query - Increased pollers on the application pods Increased application pods where worker are running Increased worker pods from 1 to 2 Increase concurrent execution size of workflows and activities to 600 Increase cache size and max workflow thread …

Hi Samar @samar We were finally able to get past the bottleneck of 250 when we increased out matching pods from 4 to 6. We could only ever go upto 200 with 4 matching pods, now able to move past and hit upto 300 with deterioration nonetheless but no bottleneck atleast when we scaled out to 6 pods. …

It seems to me like worker under provision issue. You can verify this by looking at your server side metrics: persistence_requests with tag operation=CreateTask, this metrics indicate that task fail to deliver to a poller and was written to database and become backlog. It will later be dispatched to…

Thanks for the response Yimin. @Yimin_Chen I do see peaks in those metrics during the bottleneck. Attaching screenshots of our 15-20 minute runs for your reference. We run 50,100,150,200,250,300 TPS runs for 2 mins each. This bottleneck was seen at 200 TPS at 5.36 PM - [Screenshot 2022-07-06 at …

You can increase your poller count per each worker pod (for example from 20 to 40), and see if that would move the needle.

We have tried moving it up to 64, did not change the outcome at 250 TPS. @Yimin_Chen As mentioned the time between activity executions where workflow task is executed steadily increasing at higher TPS, but workflow task schedule to start and execution latencies remain low. It’s the time after an ac…

Hello @Yimin_Chen we took a closer look at what you suggested in provisioning more pollers and also added more worker pods to reduce the async match count. What we noticed happens is even though there are 10 pods running with 50 pollers and 200 maxconcurrentexecutionsize, we dont see 10*concurrent_e…

As long as you see persistence_request {operation=“CreateTask”} it means there is no poller waiting when the task is dispatched. Did you take a look at this doc: How to tune Workers | Temporal Documentation

Seeing high latencies between two subsequent activity task executions

Community Support

Yimin_Chen July 8, 2022, 3:59am 15

What SDK do you use? Could you share your worker initialization code?

Incorrect task slots getting created in K8S multi pod deployment

Topic		Replies	Views
Workflow task timed out on GKE Community Support java-sdk , cassandra , metrics	6	1122	June 8, 2022
Temporal performance with golang microservice, Cassandra & Elasticsearch Community Support go-sdk , elasticsearch , cassandra , docker , performance	14	3561	February 1, 2023
Temporal throughput Community Support general-impl , best-practices	16	6189	January 20, 2025
Tuning Temporal setup for better performance Community Support cassandra , performance , kubernetes	5	9504	November 13, 2021
Temporal seems to hit scale wall Community Support performance	6	3721	March 29, 2024

Seeing high latencies between two subsequent activity task executions

Related topics