The issue I’m encountering involves the initiation of workflows through an API call. Despite making API calls to start more workflows, only 8 workflows are active at a time.
Could you give more info on this please? How are you deploying your service? Do you have service metrics configured that we can look at? Any other info much appreciated.
The temporal cluster is deployed in AKS. We are connecting to the cluster from our application deployed in local environment. What we observe is while initiating multiple instances at a time, maximum 8 instances are seen active in the web UI, if some of the instances are completed or terminated, remaining instances will be appeared. We are using default name space and as per our understanding we have not provided any additional configurations to the environment or namespace.
Could you please help us to figure out the reason for this behavior.
Hi @tihomir ,
Can you please help me to solve my issue with above explanation
maximum 8 instances are seen active in the web UI, if some of the instances are completed or terminated, remaining instances will be appeared
Can you look at the Start and End times of these executions once they all complete and share maybe?
There is no restriction to X number of workflow executions that can run concurrently so I am a bit confused given the current info you provided as to what is causing this still.
Can you also give bit more info on your workers? Which Temporal SDK do you use and what WorkerOptions do you set?
workflows are taking nearly 2 minutes for execution. We are using Java SDK, and following worker option is set
Worker worker = workerFactory.newWorker(CommonConstants.TASKQUEUE,
WorkerOptions.newBuilder()
.setMaxConcurrentWorkflowTaskExecutionSize(10)
.setMaxConcurrentWorkflowTaskPollers(10)
.build());
Still getting same issue. While initiating 10 workflow instances at a time, only 8 instances are created. We have executed the following CLI command is also returning same count of workflow instances listed in web UI.
tctl workflow count --query 'ExecutionStatus="Running"'
The logs created while running 10 workflows at a time:
2024-02-08T11:01:14.637+05:30 INFO 21636 --- [ task-2] i.t.s.WorkflowServiceStubsImpl : Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=2, target=<server>}}
2024-02-08T11:01:14.637+05:30 INFO 21636 --- [ task-3] i.t.s.WorkflowServiceStubsImpl : Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=7, target=<server>}}
2024-02-08T11:01:14.637+05:30 INFO 21636 --- [ task-1] i.t.s.WorkflowServiceStubsImpl : Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=8, target=<server>}}
2024-02-08T11:01:14.637+05:30 INFO 21636 --- [ task-5] i.t.s.WorkflowServiceStubsImpl : Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=<server>}}
2024-02-08T11:01:14.637+05:30 INFO 21636 --- [ task-6] i.t.s.WorkflowServiceStubsImpl : Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=3, target=<server>}}
2024-02-08T11:01:14.637+05:30 INFO 21636 --- [ task-8] i.t.s.WorkflowServiceStubsImpl : Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=5, target=<server>}}
2024-02-08T11:01:14.637+05:30 INFO 21636 --- [ task-4] i.t.s.WorkflowServiceStubsImpl : Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=4, target=<server>}}
2024-02-08T11:01:14.637+05:30 INFO 21636 --- [ task-7] i.t.s.WorkflowServiceStubsImpl : Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=6, target=<server>}}
workflows are taking nearly 2 minutes for execution
Can you explain reason for setting MaxConcurrentWorkflowTaskExecutionSize to 10 (default is 200).
I think this could be reason executions take longer than you expect as assume you have a single worker?
In sdk metrics would watch your workflow_task_schedule_to_start_latency assume its pretty high.
This to me still does not explain the “8 executions only” part, are you sure you are starting 10 executions with unique workflow ids on same namespace?
