Push, not pull, workers?

Community Support
,
1

Hey, I’m starting a new project that needs a task queue, and came across Temporal while searching for a task queue that works well with TypeScript. From reading the documentation, I’m pretty impressed, and I’d be quite interested in trying to set up Temporal with Temporal Cloud.

I have one sticking point though, everything I’m reading implies that workers are supposed to be long-running processes that poll Temporal for work. For my use-case, this is a non-starter, it’s too expensive. I’m looking at a task queue because I expect the queue to have burst/unpredictable load, and I would want to set up different queues with different throttling characteristics. If I were to choose a pull-based system, then I need to worry about scaling the pull-based workers up and down according to queue length, which isn’t desirable as it’s not really part of project value.

It would be much, much simpler for me to have a scheduling system where, when there are new tasks in the queue, the scheduling system pushes the task by spawning a worker with the task; the implementation for which could be triggering an AWS Lambda function or starting an AWS ECS task (on Fargate), depending on timeout. The worker then notifies Temporal whether it completed the task or errored.

Is this not possible today with Temporal? If not, is it on the roadmap?
Thanks

2 Likes
2

You can have a small number of workers that implement an activity that calls into lambda.

1 Like
3

If I understand you correctly, the idea is to use Asynchronous Activity Completion, where there’s a long-running worker that passes a token such that AWS Lambda / ECS can report the activity success or failure to Temporal? Therefore, while the worker does need to stay online all the time and incurs additional expense, because the worker is essentially only acting to dispatch tasks and doesn’t do any heavy computation itself, it should be able to scale quite high while remaining on a small server, such that it should never really incur a large expense in and of itself?

4

Yes, you understand correctly. Also, note that you need a workflow worker anyway, so running an activity worker on the same host shouldn’t add much overhead.