Each workflow in our system produces a final output value – typically an S3 file or a simple JSON payload. When a workflow completes, we want to post that final output to a registered callback. Similarly, we have a status API which consumers can query to also get the status and final results of completed workflows.
In the case of an S3 file, we need to generate a presigned URL for the file so external consumers can access it. The logic for this lives in a QueryMethod (getFinalResult) on the workflow class.
The status controller retrieves the execution status and uses the describe workflow execution response to determine if the workflow has completed, and if so, it calls the getFinalResult query method. I wanted to reuse this logic for the callback notification activity. However, the workflow isn’t completed at the time the activity is triggered.
Given that, my questions are:
What is a good way of handling this? I tried creating a child workflow and having it query the parent workflow for the execution status, but the parent workflow is still “running”. Can you complete the parent workflow and start a child workflow so the execution status would work?
If the above is possible, is there a problem with using the describe workflow execution method? Is there a possibility that temporal hasn’t seen the updated status at the time the child workflow would run? I wasn’t sure if those were synchronous processes or not.
I’m not sure I understand the problem. Would computing the URL and storing it in a workflow field solve the problem? Then both the activity and query method could use that value. The query method should return empty/null until the save to S3 activity is completed.
My concern is with the expiration period of the presigned URL. Theoretically we need to support the ability for someone to retrieve the final output for several months which extends beyond the limit for a presigned URL expiration period.
So the presigned URL has to be regenerated at least periodically.
Meaning implement some sort of loop within the workflow that calls the presigning activity, sleeps during the expiration period, and then iterates again?
Wouldn’t this also mean the status controller I mentioned earlier wouldn’t know that a workflow is complete without also checking for the presence of some final value within the workflow?
Note that while Temporal supports a very large number of open workflows you have to watch your DB disk utilization carefully in this case. So test for the largest possible number of open workflows to size your DB appropriatly.