Workflow and Activities with OAuth token

Community Support
1

I’m trying to solve a problem with Temporal and curious if anyone has done something similar before or might have some suggestions. I have a Workflow with a few activities running concurrently with Async/Promise which communicate with a third-party service using OAuth2. The OAuth2 handshake is completed by a web application backend and the bearer token and a refresh token are passed to the Workflow as an argument to the WorkflowMethod. (Will need to consider encryption but I think I know how to do that and out of scope for this particular problem.) When the OAuth2 token expires we will need to refresh it. However, I need to ensure multiple activities using the same OAuth2 token don’t refresh themselves. I’ll need a single activity to do the refresh and then update the state of the Workflow with the new token to then pass to the activities. I think having an activity fail with some sort of AuthenticationException should allow the Workflow to handle the exception and then invoke a token refresh activity and then either retry the failed activities with the new token or invoke new ones. One idea I’m still working through is using a timer in the Workflow to trigger the token refresh and store the new token in the Workflow state. Alternatively, we could have an activity responsible for fetching the token and refreshing it, storing the token outside of the Workflow state. This might work better in the case of Workflow restarts. I’m still thinking through this problem but curious if anyone has solved this or done something similar.

2

In the past, I’ve solved this problem by sharing the client. That is, all your activities should be using the same instance of this OAuth2 client. The client will have the appropriate interceptors to handle the token refresh lifecycle (e.g. using mutexes so that refresh is only called once where necessary). Your workflow should not have to worry about it.

3

I’m expecting to have multiple Workers running in separate processes (on different machines) so the scenario I was trying to avoid is when multiple OAuth2 clients attempt to refresh the token.

4

I’ve solved this issue by having an activity throw a specific exception indicating the token has expired. The OAuth client in the activity will not attempt to refresh the token. The Workflow will catch the exception thrown by the Activity, invoke another Activity to refresh the OAuth token, and invoke the previous Activity with the fresh token. It looks something like this.

private lateinit var currentCredentials: Credentials

override fun workflowMethod(credentials: Credentials) {
  currentCredentials = credentials
  withFreshCredentials { creds -> activities.doSomethingAuthenticated(creds) }
}

private inline fun <R> withFreshCredentials(block: (credentials: Credentials) -> R): R {
  var attempted = 0
  val attempts = 1
  while (attempted <= attempts) {
    try {
      return block(currentCredentials)
    } catch (e: ActivityFailure) {
      if (e.causedBy<TokenExpiredException>()) {
        currentCredentials = authenticationActivity.refreshToken(currentCredentials)
      } else {
        throw e
      }
    } 
  }

  throw RetryExceededException("Failed to refresh token after $attempted attempts")
}

This method probably don’t work that well with Async/Promise Activity invocations but could probably be re-worked to handle that. Still playing around with this idea.

5

hmm. Why cant you manage refreshing of tokens in the activity?
it could eitehr be a seperate activity or the first step in any activity which makes outbound calls ( this way it could be free from workflow code)and as far as retry is concerned you could possibly piggy back on the activity retry itself.

For token validity, genrally most oauth providers provide a token introspect endpoint, which can be used to validate the token, and since you anyway possess the refresh token, you can alway fetch a new bearertoken.

but if you want to pass on the new bearer token value, then you will need to persisit that into the workflow (after freshing the token and may have to return it as your activity response)

and as far as encryption is concerned, this thread has some details

6

oh, i see you have already solved it! cool!

7

I didn’t want to handle refreshing of the token in the Activity because it wouldn’t support running concurrent Activities with Async/Promise. If 2 Activities are invoked concurrently with a stale token, they will both attempt to refresh, one will fail, the other will succeed. The new token is only available in the Activity which successfully refreshed the token. The Workflow won’t know there is a new token to pass to subsequent Activity invocations. I could return the new token from the Activity which successfully refreshed the token but I’m also returning additional information and the token refresh seems like a separate concern. I could also inspect the token to see if it is expired and refresh if it is, rather than invoking the Activity and raising an exception. I might add this but the likelihood of an expired token is pretty low (though can still happen for long running Workflows) and the number of Activity invocations high. Seems like a waste to inspect the token each time. The solution I provided above does work though still trying to work through some details when using Async/Promise to run concurrent Activities. WRT to encryption, that’s a separate problem but I know how to solve that with a custom DataConverter.

1 Like
8

yup. multi threaded race condition could be a problem here… your approach seems better.