Hello,
I’m getting an error when trying to put this value in the worker builder:
opts.MaxConcurrentWorkflowTasks = 1;
This exception is thrown:
System.InvalidOperationException : Invalid options: max_concurrent_wft_polls cannot exceed max_outstanding_workflow_tasks
What else i have to do to make my worker process one workflow at a time?
Thank you!
Why do you want the worker to process only one workflow at a time?
Often when people ask this question, it’s because they have a functional requirement: there’s some series of action that the workflow needs to do, without being interleaved with actions from another workflow.
Often it’s a bad idea to attempt to implement functional requirements with operational restrictions. I don’t know if MaxConcurrentWorkflowTasks would even do what you want… it might only run one workflow task at a time, but I don’t know if that would prevent the workflow from picking up tasks from different workflows to run? Even if it did work, you’d make your cluster expensive to run since you’d be underutilizing the worker, and, I imagine, you might need to run only one worker… which would make your system less available.
For a functional requirement, often the best place to implement that is by using workflows. Route the work to be done to a specific workflow using signals or updates, and have the workflow perform the tasks one at time as you need it to do.
i see, this an excel import with more than 1 million of lines and we can’t import the same file for example if the client import twice, so we need to block this workflow to run one workflow at time. Do you have any example of how to do what you said?
Thanks in advance
Stumbled across this thread while looking for a solution myself in Python, eventually figured it out.
When defining a worker you can’t specify max_concurrent_workflow_tasks=1 for two reasons
max_concurrent_workflow_task_polls (how many tasks the worker is going to ask the temporal server for at a time) can’t be more than the number of tasks the worker is allowed to concurrently process.max_concurrent_workflow_task_polls isn’t good enough as we also have to disable the sticky queue polling. ( I believe you could set nonsticky_to_sticky_poll_ratio=0 but I haven’t tried it). Alternatively you can just disable it altogether by setting max_cached_workflows = 0 . This may have performance implications please give this a readThis handy comment explains it better than I ever can.
https://github.com/temporalio/sdk-core/blob/7b5f0ed40702b9b89bcc9cff10c393bbe4d05e3f/core-api/src/worker.rs#L44-L49
TLDR Not sure of the syntax in .Net but this config in Python works:
max_concurrent_workflow_task_polls=1,
max_concurrent_workflow_tasks=1,
max_cached_workflows=0,
this an excel import with more than 1 million of lines and we can’t import the same file for example if the client import twice, so we need to block this workflow to run one workflow at time.
you can rely on the workflowId to deduplicate the customer request, see What is a Temporal Workflow? | Temporal Documentation
If you need to manage the number of concurrent workflows, I think @awwx will work. Or similarly, you can have a single (parent) workflow from where you can start and control the number of concurrent child workflows and signalWithStart the parent workflow to add work.
@joao.zacharias If the client imports the same file twice, how do you identify that it’s the same file? For example, are you using the filename to tell whether it’s the same file, or a hash of the file contexts, or…?
Then, as @antonio.perez says, you can base the workflow id on the id that you’re using for the file, so that multiple requests for the same file are routed to the same workflow.
If the client starts an import as an asynchronous operation, you could use a signal to send the import request to the workflow; if perhaps the client is making something like an API call to perform the import and you want the API call to wait and return when the import is completed, you could use a workflow update.
Then the logic of the workflow can decide what to do if a second request comes in when a first request is still running. Do you want to reject the second request? Have the workflow respond to all requests after processing the import once? Queue the second request to perform another import operation after the first one is done? Cancel the processing of the first request and start processing the second request? However you want to handle a second request coming in while a first one is still being processed, you can program the logic of the workflow to do that.
I understand how this prevents a worker from running more than one workflow task at a time, but how does this prevent a worker from running an available task from one workflow, and then running an available task from a different workflow?
It won’t. You can use another workflow to manage/control concurrent workflows