Temporal Workflow documentation does not recommend using ThreadPoolExecutor or native threads.
But in the event am doing file processing(containing many records) how does temporal manage threads ensuring that the processing is fast and efficient. Currently am able to achieve the above using threadpool executor because am able to control resource utilization(threads) using its configurable properties.
You can use ThreadPoolExecutor in an activity implementation. Workflow should be used as a control plane for big data, not for passing large amounts of data through them.
If you described your use case in more detail, I would give a more concrete recommendation.
Cool. I am doing file processing where by am processing records from an excel file. My concerns are: recoverablity part of it(In the event the application goes down) and speed of processing.
Currently am able to achieve speed whereby am processing the records in an activity however recoverablity isn’t. How can I achieve both?
An activity can heartbeat and include the progress information into the heartbeat. If it fails on retry the value of the last heartbeat can be accessed by the activity implementation. This way it can continue processing the data set from the last heartbeat point.
1 Like
Nice,Thanks.