The workflow wake up periodically and query db for new entries
It saves the last id of the entry processed and go to sleep
After wake up it query db for new entries based on the id the workflow saved
So eventually it will exceeds history max size and we are supposed to use ContinueAsNew
I don’t mind ContinueAsNew every time the workflow wakes up, but how to pass down the last id to the new execution?
Thank you!
workflow wake up periodically and query db for new entries
The workflow should do the above logic like 100 times before continue as new
continue as new example:
You pass it as an argument to the workflow.
Thank you for the replies. I’d like to share some more of my thoughts and questions around this.
Let’s say I have a workflow for process one db row. I can start multiple execution of that workflow in parallel. I will make sure to build the workflowId (should we rename this to executionId?) with the db row id in it so I will not start multiple executions on the same row.
Now say I built a workflow which takes the row id of the last processed as argument, query db to get all the new row ids came in after. Loop through them and start execution of the workflow above for each row id. For this workflow, I want one and only one execution running. So it will have a fixed workflowId. Say I trigger it manually right now, I’ll use the same workflowId to prevent 2 running at the same time, but have to run it ContinueAsNew since the previous run has the same workflowId and won’t start if I don’t say ContinueAsNew.
Now the question is how do I convert the workflow above into a cron? Or should I create another cron workflow to just trigger it? Can a cron workflow read the result of the previous run to figure out the last processed row id and use that to trigger a new run? Or should the previous run report that number back and the cron Workflow saves it? But then again the issue is the cron workflow eventually ran out of history space as well.
From operation perspective, we need to be able to easily pause/resume/modify the cron (is code change too heavy for this?), locate runs and figure out what happened, and re-run failures after fixed data/code. I hope Temporal can provide a more complete picture around the cron story