We have the following pattern in the application we are currently building:
[incoming requests] → [instantiate a workflow for each request] → [Temporal] → Workers → [External Applications]
The external applications have a black out time based on their business hours (say, between 9PM to 6AM), in which any requests are not accepted (rejected with an error). So, even though our application receives the incoming requests, we should not send the requests from the Temporal workers to the external applications during this time and queue them up till next open window.
I could not find a direct way to stop a worker based on some time window or pause a workflow for a specific time period (based on the external app’s blackout time). I referred the following links but somehow couldn’t decipher a solution to serve my need:
What's the right way to pause/unpause a workflow with interceptor and signal?
I saw a GitHub open issue on this as well: Workflow Pause / Unpause · Issue #3006 · temporalio/temporal · GitHub
One option I tried is to check in the activity’s code if the current window is a blackout time and if so, I return an error and let Temporal workflow keep retrying it. I did not find this very elegant. Also, I have the following query:
Since we have no control on the incoming requests and for business reasons we have to accept those requests even in the blackout time window, we have to somehow buffer those until the black out window is over. With the retry based approach, the requests keep piling up in the task queues. Won’t it affect the performance and scalability of Temporal itself?
Any help on this will be highly appreciated.