Infinite loop inside a workflow

amir
Mon, 03 Jul 2023 08:59:05 +0000
Community Support

I would like to discuss the implementation of an infinite loop inside a Temporal workflow that periodically polls data from a database and starts multiple child workflows based on the retrieved data. This approach involves executing an infinite loop with a 2-second delay between iterations, initiating up to 70 child workflows in each iteration (between 0 up to 70).

@workflow.defn
class StarterWorkflow:
    @workflow.run
    async def run(self):
        max_tasks_to_continue_as_new = 500
        current_processed_tasks_count = 0
        while True:
            tasks = await workflow.execute_activity(poll_scrape_tasks)
            for task in tasks:
                current_processed_tasks_count += 1
                workflow.start_child_workflow(MyChildWorkflow.run, task, parent_close_policy=workflow.ParentClosePolicy.ABANDON)
            await asyncio.sleep(2)
            if current_processed_tasks_count >= max_tasks_to_continue_as_new:
                workflow.continue_as_new()

Questions:

1. How does the workflow handle failures in the poll_scrape_tasks() activity? Will the workflow retry if an error occurs?

2. Is this approach considered a best practice for implementing my scenario?

6 posts - 3 participants