Using temporal for scalability within an activity

Community Support
1

I have a temporal activity that gets azure datalake file path as input. Now this path can have multiple parquet files. Currently, the activity picks up one file after another, processing each file in a sequential manner. Something like below, processFile is a heavy method and it takes lot of time. Also, the no of files can be huge as well.

datalakeDirectoryClient.listPaths().stream().forEach(pathItem -> { 
   processFile(pathItem.getName()); // time taking method
});

Is there any way I can use temporal for scalability, get it to process these files in parallel. Take for eg. dynamically spawning multiple instances of activity each picking one of these files.
Or do you suggest its better to use utilities in Java SDK itself, for eg: Parallel streams

2

You have two options. The first one is to implement parallel processing from a single Temporal activity. It assumes that the processing is going to happen in a single process. Another option is to run multiple parallel activities, each activity can run in a separate process.

3

First option when you say parallel processing from single temporal activity, you mean using Java’s capabilities( Eg: Parallel streams/ multithreading/async calls wth completable future etc). Am I right?

Second option of running multiple parallel activities, could you please let me know how this can be done.
The calling workflow in our case actually belongs to another team, and they just call our activity with the directory path. From this activity, do we have any way of dynamically spinning up multiple parallel activities.

Could you please clarify Maxim.

4

Yes, the first option assumes that your activity implementation uses standard Java capabilities.

For the second option, you can switch from an activity to a workflow and ask the other team to call it as a child workflow.

5

Thanks for the response Maxim.

It might be difficult to get the other team to switch to a workflow call instead of activity. Any other way of doing it using the activity itself?

I saw this example where this download call is actually returning queue name dynamically. Suppose say we get queues dynamically for each of the files in the directory and we call the activity with different queues parallely. Is this a bad approach or does it make any sense.
Also if possible, please let me know in what scenarios we can go for the approach in File Processing sample.

6

You can call that activity yourself in parallel without relying on another team to perform parallelization.

7

Am sorry, but could you please elaborate. If this was a workflow, then I could just call activities asynchronously.
The call to my activity comes from a workflow, then how can I call activity myself in parallel. Any sample that I can refer to.

Or You mean from my activity to call a new workflow that triggers activities asynchronously.

8