Python-sdk parallel activities

[Chen_Wan]

I was wondering with the python-sdk if it’s possible to execute multiple activities in parallel. I am looking for a similar capability shown in the go-sdk example - https://github.com/temporalio/samples-go/blob/master/parallel/parallel_workflow.go.

I have tried prototyping using https://github.com/firdaus/cadence-python with Cadence and hoping when temporal python-sdk is already, I can make the switch. So far I am not able to run the activities in parallel. I have tried wrapping activities in asyncio tasks and using asyncio.gather to wait, but the activities don’t execute.

I have a use case for downloading multiple large VM images. I am hoping to have multiple download activities running in parallel to finish faster. Go and Java sdk clearly supports that kind of use case. It’d be nice to accomplish the same with python sdk.

[firdaus]

Not yet supported in cadence-python. See “Parallel activity execution” in the roadmap.

Using asyncio primitives won’t work because the activities need to be started in parallel using Temporal’s scheduling features.

Current focus is on porting existing cadence-python features to temporal-python. Once that is done this feature is quite low hanging fruit and lots of people have been asking for it, so i’ll definitely prioritize this.

[Chen_Wan]

Thanks for the prompt response. Good point on the roadmap, I missed it somehow.
Yeah, it makes sense to port the existing features first. I understand that you’re working hard on that and your efforts are definitely much appreciated.

[firdaus]

@Chen_Wan

This is still work in progress but let me know what you think of this:

• https://github.com/firdaus/temporal-python-sdk/blob/master/tests/test_activity_async_any_of.py
• https://github.com/firdaus/temporal-python-sdk/blob/master/tests/test_activity_async_all_of.py

[Chen_Wan]

@firdaus. This looks terrific. Thank you!
I run a quick test using your tests and it works well.

I was wondering about how the activity timeout would work with the Async timeout. Does the Async method’s timeout value override the activity timeout? Also, curious about how retries are handled?

[firdaus]

For timeouts you can:

• any_of() and all_of() both support a timeout
• in the decorator - https://github.com/firdaus/temporal-python-sdk/blob/5e894a858c0fb166f56bb0669dc168def7200c47/tests/test_activity_method_activity_options.py
• in the options in the activity stub - https://github.com/firdaus/temporal-python-sdk/blob/2675b37820636689b947eb68198464013bda6aca/tests/test_activity_method_activity_options_from_stub.py

For retries, you can use the normal retry parameter support:

• https://github.com/firdaus/temporal-python-sdk/blob/f77dae844969fa40937bd70d407c35af14e2a318/tests/test_activity_retry_maximum_attempts.py
• you can pass the retry parameter in the decorator as well - https://github.com/firdaus/temporal-python-sdk/blob/5e894a858c0fb166f56bb0669dc168def7200c47/tests/test_activity_method_activity_options.py

[Chen_Wan]

Nice and clear examples. Thanks.