Load Testing on Temporal. Has anyone done a basic load testing on Temporal project?

Community Support
, , ,
1

I have deployed my Temporal setup ( temporal server using helm and a workflow microservice) on aws eks. So I have been trying to load test that using Jmeter.
Have we done that already to have a benchmark ? How many requests can be handled by temporal ?
After some time I have started getting

2022-02-14 07:55:52.303 WARN 1 — [l-system": 2062] io.temporal.serviceclient.GrpcRetryer : Retrying after failure
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9.999983763s. [remote_addr=172.20.160.56/172.20.160.56:7233]
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.38.0.jar!/:1.38.0]
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.38.0.jar!/:1.38.0]
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.38.0.jar!/:1.38.0]
at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondWorkflowTaskCompleted(WorkflowServiceGrpc.java:2654) ~[temporal-serviceclient-1.0.9.jar!/:na]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.lambda$sendReply$0(WorkflowWorker.java:383) ~[temporal-sdk-1.0.9.jar!/:na]
at io.temporal.serviceclient.GrpcRetryer.lambda$retry$0(GrpcRetryer.java:77) ~[temporal-serviceclient-1.0.9.jar!/:na]
at io.temporal.serviceclient.GrpcRetryer.retryWithResult(GrpcRetryer.java:95) ~[temporal-serviceclient-1.0.9.jar!/:na]
at io.temporal.serviceclient.GrpcRetryer.retry(GrpcRetryer.java:74) ~[temporal-serviceclient-1.0.9.jar!/:na]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.sendReply(WorkflowWorker.java:376) ~[temporal-sdk-1.0.9.jar!/:na]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:323) ~[temporal-sdk-1.0.9.jar!/:na]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:279) ~[temporal-sdk-1.0.9.jar!/:na]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:73) ~[temporal-sdk-1.0.9.jar!/:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

Can someone please help me with either the error or the load testing ?

2

Temporal provides the Maru load simulator which is specific to workflows written using the Go SDK.
As for scalability, there are production use cases which have hundreds of millions open workflows at any time.

From your error it seems that workflow workers either cannot connect to Temporal service, or are having issues handling the test loads. Would recommend this forum post for a Java SDK specific worker tuning tips.

2 Likes