I’m evaluating Temporal for a project, and wanted to clarify a couple of things, and get your opinion if it’s a good use case for Temporal.
Am I understanding this correctly:
I can create a number of Worker services, and each of those Worker services will only run Activities and Workflows that I will register for them.
So say, that I want to allocate part of my k8s cluster nodes to execute only specific types of tasks, I can just register a single workflow and activity within those specific workers?
It is also mentioned that:
If the worker receives tasks for a Workflow type it does not know, it will fail that task.
I assume, that I control if a specific Workflow-task will be delivered to a specific Worker-service using Task Queue names?
So I can avoid the situation of auto-failing tasks, just by using correct task queues?
Now to the use cases:
Video/image processing - a media file is uploaded to a server, and it needs to be processed - converted to a different format, sliced, down/up-scaled and so on. So, an operation that can be very long running. There is a limit to the concurency, due to limit of the availiable hardrware, so if all the processing nodes are busy, new processing tasks should just queue up.
It’s also mentioned that:
A Worker polls for a message only when it has spare capacity, so it never gets overloaded.
Can this be controlled somehow? Say I want to limit a worker to execute only one task at the same time.
Async fan-out - a way to propagate, when one event results in thousands of insert/delete queries to the DB. Especially popular within the Cassandra community.
Do you think Temporal is a good fit for these two?
So I can avoid the situation of auto-failing tasks, just by using correct task queues?
Yes, use task queues to route requests to the appropriate worker pools (aka services).
Video/image processing
This is pretty common use case for Temporal. See the fileprocessing sample which demonstrates it.
Can this be controlled somehow? Say I want to limit a worker to execute only one task at the same time.
The best way to achieve this is to use Go SDK Sessions feature. It allows limiting a number of parallel sessions running on a worker/host. Each session can be composed of multiple activities.
Async fan-out
Would you elaborate more on the specifics of this use case. Is it running some updates based on some externally generated list of IDs?
Awesome, thanks for a quick reply, will look into those links
An example would be something like a social network interaction. Each user has a list of friends. Each friend has an individual feed of updates from other friends. When a user makes a post, that postId should be inserted into the feeds of all the friends.
So, you have a list of userIds, and you run a for-loop to insert postId into each one of those users’ feeds.
Different approaches can be taken with Temporal as I see it - it can be breaking a large set of userIds into smaller sub-sets and spawning a new workflow for each one of those sub-sets.
I see. In this case, the simplest solution is to send notifications from an activity that iterates over the data set and executes the update operations. To ensure timely restart the activity should heartbeat including progress information as heartbeat data. If the worker that executes the activity fails the heartbeat stops, and the activity is retried. Then it loads the progress information from the last heartbeat data and continues.
