We’re building a new application to process data from a diverse, distributed set of data sources. For simplicity’s sake, let’s assume we have a single workflow to process data, and a set of connectors which abstracts the differences between the different data sources, i.e. each connector implements a common interface, i.e. a common set of activities. The connectors need to be deployed close to the data sources, which can be distributed across different geographies.
We’re planning to run each connector as a separate Temporal activity worker, with a central workflow worker executing a single workflow per connector. The workflow needs to route activity executions to the correct connector. We are planning to use the TypeScript SDK to implement the workflow(s).
I’m sure I’ll have more questions, but for now I’m trying to figure out how to manage the dynamic routing of activities to the correct worker. A similar concept – though for a different purpose – was discussed in Using dynamic task queues for traffic routing. A feature request for dynamic task queue routing was filed by @maxim but has so far not been picked up.
I was wondering if it’s feasible/advisable to use a much simpler solution like this:
import type * as connector from ‘./connector/activities’
export async function processData(connectorId: string): Promise<void> {
const connector = proxyConnector<typeof connector>(connectorId)
await connector.fetchData();
// … subsequent processing steps
}
function proxyConnector<A extends ActivityInterface>(connectorId: string): A {
return proxyActivities<A>({
taskQueue: `connector-${connectorId}`,
// other parameters
})
}
In all the samples, I’ve only see the proxyActivities function being used outside the workflow to create a static set of activity proxies. In a simple PoC, I’ve been able to use this function in the workflow, as per the example above. But I wonder if this is intended usage or whether it might create suble (or not so suble) problems down the road?