Infinite loop inside a workflow

Community Support
, ,
1

I would like to discuss the implementation of an infinite loop inside a Temporal workflow that periodically polls data from a database and starts multiple child workflows based on the retrieved data. This approach involves executing an infinite loop with a 2-second delay between iterations, initiating up to 70 child workflows in each iteration (between 0 up to 70).

@workflow.defn
class StarterWorkflow:
    @workflow.run
    async def run(self):
        max_tasks_to_continue_as_new = 500
        current_processed_tasks_count = 0
        while True:
            tasks = await workflow.execute_activity(poll_scrape_tasks)
            for task in tasks:
                current_processed_tasks_count += 1
                workflow.start_child_workflow(MyChildWorkflow.run, task, parent_close_policy=workflow.ParentClosePolicy.ABANDON)
            await asyncio.sleep(2)
            if current_processed_tasks_count >= max_tasks_to_continue_as_new:
                workflow.continue_as_new()

Questions:

  1. How does the workflow handle failures in the poll_scrape_tasks() activity? Will the workflow retry if an error occurs?
  2. Is this approach considered a best practice for implementing my scenario?
2

It will raise which can be caught, but in your code where it is not caught, it will fail the workflow. You can set a workflow retry policy for it to retry but it doesn’t by default.

Yes, this can work. There are other common approaches for polling in Temporal. See this sample which shows 3.

1 Like
3

Thank you for your response. I did take a look at them, but unfortunately, they don’t address the specific requirement I have, which is starting workflows or child workflows based on the polled data.

In my scenario, I need to initiate multiple child workflows or workflows for each data item retrieved from the database. This cannot be accomplished through activities alone (starting workflows through activity is not possible i think) and requires to be done on workflow code.

4

Yes, if they must be child workflows, what you have technically works. What many do is start regular top-level workflows from inside the activity as needed using a Temporal client. It sounds like in your case they don’t need to be child workflows (you don’t wait on them even starting successfully and you “abandon” them for cancel/complete), so top-level workflows started from the activity will probably work best for you. Then you can use one of the first two polling approaches that uses activities but also start a workflow in that activity (just make sure the activity is idempotent).

1 Like
5

Thank you for your prompt response

6

Have you thought of possibly moving to using a Schedule-based approach? It could be more robust but it would require updating to 1.20.