I have a parent workflow, “MigrateFileDataWorkflow,” that needs to handle around 200K UUIDs. For each UUID, I need to call an external service async grpc API thats trigger the download of a file linked to UUID.
To manage this, My thought process is from parent create and call asynchronous child workflow, called “DownloadMetaDataFileWorkflow,” for each UUID. This workflow will then call the “DownloadMetaDataFileActivity.” that invokes an asynchronous gRPC API to download the file, which is idempotent API, this activity also need to periodically poll another gRPC API to check the download status.
However, I am concerned about the limitation of 50,000 history events per workflow and also the maxWorkflowThreadCount that has default limit of 600. With the above approach, we will be creating approximately 200,000 child workflows and a equal number of activities.
while working on a POC for 1000 UUIDs i’m getting the error
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6f1d97c5 rejected from java.util.concurrent.ThreadPoolExecutor@f5fbe64[Running, pool size = 600, active threads = 600
I would like to confirm if my understanding is correct and if there are any better options available for handling this situation, using temporal java-sdk