I’m designing a web crawler, and one feature I’d like to implement is a per domain rate limit, that’s separate from the existing workflow or activity rate limit and configurable dynamically.
I realise I could deploy a new worker for the domain however as a single worker will be able to make many thousands of outbound HTTP requests, I’m looking for alternatives to this.
In the past I’ve used a throttler via middleware within the HTTP transport to rate limit per-domain, perhaps this could be implemented via WorkflowInterceptors?
Is there an ideal “Temporal” approach here? I think I’m looking for a per-domain task queue rate limit but I’m not sure if it’s possible or how to achieve it using the Go SDK.