With Temporal we managed a workflow splitted on 3 phases (on Java SDK):
- Loading phase: extract from oracle source of all datas to manage. I save it on ms with mongodb
- Engine phase: read from mongodb populated data and manage the elaboration(read/write operation, huge dataset)
- Extractor phase: read result and copy it on oracle, to close the circle
In technical point of view, Engine phase will start only when Loading phase finished. Engine is the part involved on our performance insight.
Atm we have an helm instance of Temporal (default values, just 1 replica) deployed on k8s (fe, web, worker, history and matching)
Temporal’s params are:
- maxConcurrentWorkflowTaskExecutionSize: 1000
- maxConcurrentActivityExecutionSize: 1000
- maxConcurrentActivityTaskPollers: 10
- maxWorkflowThreadCount: 180000
Engine works in parallel mode, manage datas grouped by and read configuration directly from yml file. We run a lot of activities in this parallel way. (Async - 100 parallel - he needs a lot of memory to do that)
Working with params(example put 500 parallel instead of 100), the problem that i found is context deadline exceeded.
I read a lot of topic on forum that contains ur answer but i didn’t find a solution for my case.
Another importat thing is that i don’t see any correct value on grafana. I followed the guide on blog but i don’t reach the goal also for this.
I need a technical help to improve performance of Temporal to use it for each product in our company.
Thx a lot guys.