Hello, I am considering using temporal. In our set, I will have a shared worker which will run multiple workflows. I am trying to understand how does temporal worker initializes these libraries in both Python and Golang. What happens if there needs to be a network call in order to initialize these libraries? I can see that temporal worker has some options to limit how many libraries it loads? From the forums, I read that temporal workers are ‘bundled’ at the start with libraries. Does that meant that when we start the worker then temporal copies these libraries on worker pods and loads them. In my case, where I am trying to deploy multiple workflows on same worker, it will lead to 100s of libraries loaded in memory which is not desirable.
A worker is just a program you build/package/deploy yourself. What makes it a worker is that you include the temporal SDK library along with any others you need as part of your build/package/deploy methods. Temporal is not in the business of loading libraries for you at all, to my knowledge. Hope that helps!
I am not sure if I fully understand the response. If I am starting my worker like this
worker = Worker(
client,
task_queue='my-queue',
workflows=[workflow_1, workflow_2],
activities = [activity_1, activity_2, activtity_3]
)
How will temporal worker memory and compute resource requirement change if when there is no activity running on the worker. Will temporal worker resource requirements increase if we configure a shared worker with 1000 workflows. I am not considering resource requirements when workers are actually running an activity because that would be dependent upon activity logic. But what about the case when no activity is running
@maxim can you share your thoughts on whether its good idea to do this way or should be consider only few workflows per worker
Registering a Workflow/Activity probably has some basic non-zero cost since the Temporal cluster probably needs to track the inventory of available workflows/activities, but I would imagine this is fairly small.
It seems you may be more concerned about the notion of MaxConcurrentWorkflowTaskExecutionSize (and friends), which is a worker tunable. This governs how many simultaneous tasks may be scheduled on that worker at a time, and tuning it would impact peak resource consumption. There is some caching under the covers that means an idle worker may still consume some resources based on previous executions…however, I would expect this to scale with MaxConcurrentXX, not the total number of workflows defined. Perhaps a Temporal insider can add color here.