I assume I can write an activity worker that just invoke an AWS lambda and proxy the input to the lambda result back. But my goal is to not think about scaling. Can async activity help? Can the worker I wrote kick off the lambda and move on to the next thing in the task queue without waiting to kick off the next lambda?
Is there a plan to support lambda natively? Like AWS step functions
AFAIK nobody has done this yet. I’m very interested in the possibility of integrating directly with Lambda + some other service. As I think you identified, one issue is that Lambda is an ad-hoc model and our workers currently poll on the taskqueue for tasks.
There are some other limitations with Lambda that might make things harder:
Overall I think Lambda is incredibly convenient compute model and I’m very dedicated to supporting it natively at some point.
You need to decide if backpressure is something you care about. Async activity would allow invoking lambdas without limiting number of parallely running activities in any way. I would advise against such approach. For example if for some reason activity requests created a backlog in a task queue then worker is going to consume them very fast and start a lot of lambdas which might cause all sort of issues with downstream systems they will be calling into.
If I were doing such integration I would invoke Lambda synchronously from an activity to make sure that flow control is in place. You can configure pretty large number of parallely running activities (especially in Go were number of goroutines is practically unlimited) to account for possible spikes in traffic.
Thank you Maxim. Just want to clarify “configure pretty large number of parallely running activities” meaning have a lot of activity workers or something else?
A single Go worker can have hundreds of thousands of goroutines. So the number of workers will be defined mostly by the rate of execution, not by the number of parallel activitites.