Vadim
December 13, 2022, 4:01pm
1
Hi all. I want to use temporal to upload a large files using the map reduce algorithm to parse files in parallel. In pseudo-code it looks like this
var uploads int
for _, offset := range offsets {
workflow.Go(ctx, func(gCtx workflow.Context) {
activity.UploadChunk(gCtx, offset)
uploads += 1
})
}
_ = workflow.Await(ctx, func() bool {
return err != nil || len(offsets) == uploads
})
It is safe to increment the uploads
counter inside workflow.Go
or i should use the atomic
package?
Yes, only one coroutine runs at a time in a workflow, so there are no concurrency concerns. It is safe to increment that variable.
Vadim
December 13, 2022, 5:37pm
3
Thank you. In that case, does it make sense to use workflow.Go
instead of calling workflow.ExecuteActivity
in the loop?
It’s basically the same thing. What many people do is call execute activity in the loop collecting futures, and then another loop to get results. This allows activities to run in parallel. Of course workflow.Go
does too.
1 Like
maxim
December 13, 2022, 10:21pm
5
I recommend ExecuteActivity if you need to do a single parallel action. But if you need to execute multiple sequences of actions in parallel, then Go is much simpler.
See the splitmerge-future and splitmerge-selector samples for the Future based parallelism approaches.
See the goroutine sample for running sequences in parallel.
1 Like
Vadim
December 14, 2022, 5:22am
6
Thanks. I started the thread with the goroutine example, but thought about data race, but in the reply Is workflow.Go safe for concurrency? - #2 by Chad_Retz replied that this is not a issue in the case of temporal.