How should activities be mapped to task queues?
Unique task queue for each activity?
Similarly for workflows?
I remember Maxim talking about a single task queue for all workflows.
Same for activities?
What exactly should be the reasoning behind this decision?
More task queues = more memory consumption and so on?
Similarly, what should the no. of worker threads look like?
Temporal supports rate limiting for task queues (this means its rate limiting across multiple worker processes that listen on a specific task queue).
Typically if you need to rate limit a specific activity you would have a task queue dedicated for that activity invocation/execution and set your rate limits on all activity workers listening on that task queue via WorkerOptions → MaxTaskQueueActivitiesPerSecond.
I want your technical opinion on something I’m following.
For me, I have a task queue per Workflow definition impl. Is it a good idea or a bad idea?
What are the honest drawbacks of this?
I know Maxim has recommended a single task queue to start off. But my problem is I just have a single WorkflowInterface. All of my different workflow definition impl extend this. And since a task queue cannot have multiple impls for the same interface, I’m forced to have a task queue per impl.
Will it scale? What problems am I likely to face? Sandbox/QA testing seems to be working perfectly well.
What is the use case for using a single WorkflowInterface which implies a single workflow type?
The common signalling, querying, state maintenance, logging and audit code resides in a single impl which extends the base workflow contract. Even the common parent child comms reside there.
All the business logic reside in delegate classes triggered from the base impl using reflection. The delegation logic also resides in the common impl.
The workflow specific impls are empty.
I’ve ended up developing my own parent child comms due to lack of proper knowledge/understanding of the one provided by Temporal OOTB.
Hence, the lack of need for additional contracts. I can do everything I want with a single contract.
There is no need to use a single workflow type in your case. You can have a base interface that defines a common interface and then have an interface per workflow type that extends that interface. See Workflow Interface Inheritance.
Not just that, it’s all about how I’ve designed the APIs to start and instruct workflows around them.
All for them work with the single interface. My point is I don’t need additional interfaces since all my required methods are satisfied by the base interface.
Making separate (mostly blank) interfaces for each flow might just be additional work (and lot of refactoring at this stage in the game just before the release).
Having said that, let’s say I go with task queue per workflow type, I’ll end up with 30-35 queues (for now) and maybe upto 100 in the future.
What impact would it have on the architecture esp. w.r.t scaling?
and lot of refactoring at this stage in the game just before the release
If you are that close to your release and there is no time to make changes then I think you have to go with what you have, and over time consider updates to make deployment simpler for yourself.
It seems that for your chosen impl you would have to deploy and scale 30+ worker processes (which you can factor by x depending if you for example have replication set on your worker pods).
Personally I don’t think that server wise this would impact your performances more under high load compared to a single or a smaller number of task queues (but I am sure you did load testing to prove this to be true/false). Do think that with the chosen approach you might have a complex deployment scenario that you can think of simplifying short/long term.