Best practive for notifying completion of async activities

Community Support
,
1

We have a use case where consumers need to be independent of using the temporal for running the long tasks and they have to notify OR workflow needs to fetch status. I want to see what will the recommended way out of these 2

  1. Service A using temporal creates an async activity and exposes an HTTP API where consumers can call the API that will complete/fail the async activity.
    OR
  2. Service A creates an activity and sends the activity id to service B. Service B then exposes HTTP endpoint which can be polled by Service A to check the completion/failure of the activity.

Foundation architecture of temporal seems to be using the first approach where consumers need to update the status of the jobs. Would love to know if there are any known pros and cons of these approaches.

P.S. Not sure if we have any other better apprcoch to solve this use case with temporal

3

I take it that the work of this Activity happens in Service B, and the work takes too long for a synchronous call like this?

async function myActivity() {
  await result = serviceB.doWork()
  return result
}

In these cases, I’d recommend considering options 1 and 2 before the options you described (3 & 4):

1. Service B runs a Worker

  • The Workflow in Service A starts activityFoo on taskQueue: service-B.
  • Service B runs a Worker that listens on taskQueue: service-B and only registers/handles activityFoo.

Pro: Simplest

2. Service A starts the Activity, Service B completes it

  • The Workflow in Service A starts activityFoo
  • The Worker in Service A picks up activityFoo
  • The Activity does:
async function activityFoo() {
  const info = Context.current().info
  const idempotencyToken = getIdempotencyToken()
  await serviceB.startWork({
    fullActivityId: {
      activityId: info.activityId,
      workflowId: info.workflowExecution.workflowId,
      runId: info.workflowExecution.runId,
    },
    idempotencyToken,
  })
  throw new CompleteAsyncError()
}

// Note: there are different ways to create idempotency tokens, depending on how you 
// run your Workflow—whether it can be retried, Continued As New, or restarted by a Client.
// This implementation may not be what you want.
function getIdempotencyToken() {
  return Context.current().info.workflowExecution.runId + Context.current().info.activityId
}
  • Service B uses AsyncCompletionClient to heartbeat, and when it’s done with the work, does:
await asyncClient.complete(fullActivityId, "Job's done!")

Con: You have to pass the fullActivityId and create an idempotency token.

3. Service A polls Service B

  • Service A runs the Activity, which first tells Service B to do the work, and then heartbeats and polls Service B until it’s done. Something like:
async function activityFoo() {
  const info = Context.current().info
  const idempotencyToken = getIdempotencyToken()
  await serviceB.startWork({
    fullActivityId: {
      activityId: info.activityId,
      workflowId: info.workflowExecution.workflowId,
      runId: info.workflowExecution.runId,
    },
    idempotencyToken,
  })

  while (true) {
    Context.current().heartbeat()
    const response = await serviceB.getStatus({ idempotencyToken })
    if (response.completed) {
      return response.result
    }
    await Context.current().sleep('1 min')
  }
}

Con: In addition to #2 con, there’s more code and network requests.

4. Service B reports completion via API call

  • Service A runs an Activity that includes serviceB.startWork(), as in scenario #2
  • Service A runs an API server with a /complete/{runId}/{activityId} endpoint (or equivalent gRPC method)
  • When Service B is done with the work, it calls the /complete endpoint
  • The Service B API handler uses the AsyncCompletionClient to complete the Activity

Cons:

  • The API server is added complexity.
  • No heartbeating, which means Temporal won’t know when Service B stops doing the work—Temporal will have to wait until the scheduleToClose or startToClose timeout to know it failed. And Service B won’t know if the Activity gets Cancelled. So ideally Service B would heartbeat, but if it’s going to do that, it might as well report completion as well, in which case we can do scenario #2.
1 Like
4

Updated my last post with your scenarios and pros/cons!

5

I’d be hesitant to recommend this implementation for getting an idempotency token.
A workflow can run multiple times with the same workflow ID, and the activity ID typically is an incremental sequence number generated per run.
There’s no one-size-fits-all idempotency token generation method.

Note that you’ll need to uniquely identify the activity with either a task token or the workflow and run IDs (all found in the activity info).

1 Like
6

@loren Thanks for the elaborative answer. Solutions 1 & 2(mentioned by you) are better but those force all the consumers to use temporal. We don’t want to force it on consumers so we probably we will go will solution 3.

In Solution 3, We were thinking to twek the solution and instead of using 1 async activity with a while loop, we can use 2 sync activities

  1. first actvity that calls serviceB and starts the work.
  2. Another parallel activity with shorter retry times (equivalent to sleep time in your example), will work as a poller for the completion status. This activity will fail if the completion status is not returned in specified time duration after no of retries.

Do you see any caveats using this alternative solution?

(Was referring to this thread on infrequent pollers)

@bergundy Thanks for pointing to the caveats. We will also include the runID while generating the IdempotencyToken. Similar suggestion I found in the doc here

7

Do you see any caveats using this alternative solution?

Sounds good! I see Maxim recommended that for polls of every minute or slower.