I am developing an API server using Temporal. Currently I have the API service in one process (using Temporal workflows/activities in its endpoint business logic), and the Temporal worker in another process so as to allow the workers to be scalable independent of the rest of the app.
I was wondering if it’s possible to control what happens when the API service attempts to kick off a Temporal workflow/activity when the worker processes are all down, say in the event of cycling pods or something. In my local testing at least, the request simply hangs forever, presumably on the workflow execution attempt. I didn’t see a timeout option for time between 1) event getting on the task queue and 1) worker pulling the event off. Would my only option be to leverage WorkflowExecutionTimeout/WorkflowRunTimeout?
when the API service attempts to kick off a Temporal workflow/activity when the worker processes are all down
Workflow executions are created by the Temporal service. On client request your execution would be created and persisted and service would put the first workflow task on the task queue for your worker(s) to pick up and process.
If this task is not picked up by your workers until the set WorkflowOptions Run/Execution timeout your execution is going to be terminated. If you don’t specify these timeouts the exec would be able to stay in Running state “forever” until your workers come back up to resume progress.
the request simply hangs forever
I assume this is because of blocking WorkflowRun.Get, for example: err = we.Get(context.Background(), &result)
Not sure if it helps but you could create context.WithTimeout and pass that to the Get call to not block forever and could then try again after some time when your workers are back up.