My use case is a typical one where a user will have n no of workflows and each can be executed indefinitely.
As of now, I am planning to create a namespace for a user. Is this the right approach to use namespace?
What is the ideal and best way to create workers?
1). creating a worker per namespace
2). creating a worker per workflow
3). creating a worker per workflow execution
Among three, which one is the ideal/best way?
Please suggest if any other approach can be possible?
I assume that each user will have n workflow instances, right? Not n custom written workflow types?
In this case, I would recommend using a single namespace and a single set of workers. I’m not sure why would you need multiple namespaces in your case.
We will have a single Workflow Type which will read input from user, based on the input will call activities either sequentially or parallelly. Input is basically a list of java objects where a type is mentioned to call an activity can be sequential or parallel.
We will provide a UI screen to drag and drop activities to form their own workflow. we will convert this graph into json string and store it in a file, when the use triggers an execution we will read from the file and form a java object which can be executed in our Workflow.
Here, each user can log into their own account and can create their own workflow through UI and execute it. each user will have their own workflows, which is not shared.
Since workflow type needs to be defined statically we will be having a single workflow type and based on the input we will be orchestrating it.
What am doing is correct?
Yes, this sounds very reasonable. In your case, a single namespace and a single pool of workers is enough.
Okay, what do you mean by a pool of workers?
How to create a pool of workers in java?
As of now, In my dev machine, I am having a single temporal server, not a cluster. Kindly tell me how to create a pool of workers for a single server and cluster?
This is DSL workflow, right?
Okay, what do you mean by a pool of workers?
How to create a pool of workers in java?
I mean the pool of worker processes. No need to create any pools in a single Java process.
You want multiple worker processes for availability and scalability.
This is DSL workflow, right?
Right.
Does a single worker will poll/handle all of my activities and workflows. In my use case, I will be having very high no of workflow execution frequency
Thanks
A single worker per process, but you can have multiple worker processes for scalability.
In my use case, I will be having very high no of workflow execution frequency
What rates are you targeting? And how many operations per workflow?
Adding more machines is vertical scaling right ? It is not desirable.
we will be calling operations as state changes on our side. A workflow can have 1000 state changes which mean it can have run 1000 activities at most. This number is quite low.
But a user can run 25,000 execution per minute. In that case, it will be quite large. But those rates are tentative. let me check with my manager and post here.
A doubt that a worker will be polling for the tasks. Will it do any operation in between receive and dispatching a task. because I never faced the timeout ScheduleToStart timeout.
When will it arrive? only when worker outages? If yes, when/how a worker outage will happen?
Adding more machines is vertical scaling right ? It is not desirable.
I believe vertical scaling is done by making machines more performant. Adding more machines is horizontal scaling. And the best way to scale Temporal is by horizontal scaling.
we will be calling operations as state changes on our side. A workflow can have 1000 state changes which mean it can have run 1000 activities at most. This number is quite low.
But a user can run 25,000 execution per minute. In that case, it will be quite large. But those rates are tentative. let me check with my manager and post here.
This gives up to 416000 activity executions per second which is a pretty significant load. It is possible to achieve, but it would require some work to provision everything correctly. And such load is only possible with a large Cassandra cluster.
Kindly explain this?
In which use case we will use different namespace and task queue for worker to poll its tasks
Don’t set the ScheduleToStart timeout. It is needed in a very narrow set of scenarios related to routing tasks to specific hosts. If not set it is never going to expire tasks in the queue. If you need to ensure that an activity cannot execute longer than some time then use the ScheduleToClose timeout.
In which use case we will use different namespace and task queue for worker to poll its tasks
When you have multiple different applications using the same backend service.
I’m exploring namespace strategies for multi-tenant architecture and need advice on balancing between multiple namespaces per user or a single namespace. The main concerns are ensuring resources aren’t monopolized by eager customers in a single namespace solution and for multi-namespace strategy provisioning workers effectively is a big question(what do you suggest?! like k8s worker pods per namespaces?!). What are your recommendations for this scenario?
I would rate limit per customer in the service that fronts Temporal. We are considering adding features that would make this seamless in the future.