I’m investigating whether we can leverage Temporal Cloud for our workflows or not, and I’m running into one major issue: It seems the only integration path with the Temporal cluster, whether it be self hosted or Temporal Cloud, is to have some always on compute running worker tasks that are doing a long poll of the task queues held by the Temporal cluster. I’m not seeing any way to have the Temporal cluster call out to some endpoint, or send out some event, to indicate it has work that needs to be performed by a worker.
Our current architecture is serverless on AWS, using things such as AppSync, Event Bridge, Step Functions, Lambda, and Batch among others. Currently, all of our compute is on demand, so it scales to 0 when not in use. For example, a GraphQL Mutation kicking off a Step Function to drive some kind of asynchronous workflow, or a webhook event from an external SaaS product coming into Event Bridge and being routed to a Lambda to be handled. Scheduled tasks currently use Event Bridge Scheduler to kick things off.
Ideally I would like to have the Temporal Cloud tell me when work needs to be performed, via something like an event, so I can run workers in response to that. I see there are some metrics used for scaling workers as an option also, but I don’t see a way to get those metrics out of Temporal without also having some always on compute polling those metrics.
I saw this old topic about using Lambda as an activity worker, and potentially natively supporting AWS Lambda at some point, but I can’t find any information about that elsewhere: Is it possible to use AWS lambda as activity worker?
Is the integration pattern I’m seeking possible with Temporal Cloud?