Hi, Currently we are using temporal in a project, however we are not able to meet the performance required. Kindly need your help to check whether we are using the right settings/recommendation. Here are our case: We are using go. Just 1 microservice with 18 activities. Each activity only do s…

Do you have both SDK and server metrics set up? It would help figure out latencies, especially on your worker side, see here for more info. name: NUM_HISTORY_SHARDS value: “512” This seems to use the default 512 and is often too low, see here for general recommendations.

The temporal server services use docker image temporalio/auto-setup:1.16.2 Don’t think auto-setup is recommended for production deployments, see here for more info.

Don’t think auto-setup is recommended for production deployments, see here for more info. Hi Tihomir, Is there any performance drawback if we use auto-setup docker image? We use auto-setup to simplify our POC (Proof of Concept).

Hi Tihomir, I already tried using temporaio/server:1.16.2, and setup cassandra & elastic myself (create tables & indexes). But I got error below. “error”:“Unable to decode search attributes: invalid search attribute type: Unspecified” {"level":"error","ts":"2022-05-24T09:07:13.104Z","msg":"Fail…

Is there any more configuration that can be configured to improve the throughput? One thing you can check is your persistence latencies. Try Prometheus query: histogram_quantile(0.95, sum(rate(persistence_latency_bucket{}[1m])) by (operation, le)) for CreateWorkflowExecution, UpdateWorkflowExec…

Thank you for the insight, Tihomir. I already checked at resource exhausted, and found out there were RpsLimit in matching service. I did change the rps value in dynamic value and make some improvements, but sometimes there were another RpsLimit occurred in frontend service. [image] Here are t…

Have you considered using local activities for short calls?

try to use 3 frontend pods; 3 matching pods; 3 history pods; 1 worker pods; check CPU / mem utilization of above pods when running the load make sum(rate(poll_success_sync[1m]))/sum(rate(poll_success[1m])) as close to 100% as possible by increasing the MaxConcurrentActivityTaskPollers and MaxConcur…

@maxim Have you considered using local activities for short calls? Hi Maxim, We haven’t considered local activity because initially eventhough the activities are short calls, we want to leverage temporal retry mechanism to keep simplicity in our workflow code. What I read during using local act…

[image] Hilman_Adam: What I read during using local activity, we need to implement retry by ourselves and need to consider workflow timeout boundary for that particular activity. This is not correct. Local activities do support retries and their retry options can be changed through LocalActiv…

Temporal performance with golang microservice, Cassandra & Elasticsearch

Community Support

tihomir February 1, 2023, 6:34am 15

Hi @Hilman_Adam answered in this post

Topic		Replies	Views
Workflow task timed out on GKE Community Support java-sdk , cassandra , metrics	6	1131	June 8, 2022
Temporal throughput Community Support general-impl , best-practices	16	6276	January 20, 2025
Seeing high latencies between two subsequent activity task executions Community Support java-sdk , cassandra	22	3244	July 19, 2022
Tuning Temporal setup for better performance Community Support cassandra , performance , kubernetes	5	9584	November 13, 2021
Settings / Recommendations for orchestrating microservices with temporal and MySQL Community Support go-sdk , mysql , helm , performance	10	2991	February 23, 2021

Temporal performance with golang microservice, Cassandra & Elasticsearch

Related topics