Temporal Retry Async Activities

Community Support
,
1

Hi all.

I would like some advice on the following use case.
I have a workflow with the following shape:

fun worklowMainMethod(): Any {
    val userResponse = createAndWaitForUserResponse() // child workflow

    createInvoice(userResponse) // activity that sends message to kafka to serviceA
    val kafkaResponse = awaitForKafkaResponseWorkflow() // child workflow
    storeResponse(kafkaResponse) // db activity

    val response = sendEmail(kafkaResponse) // activity that sends email
}

The awaitForKafkaResponseWorkflow is a child workflow that waits for a signal, this signal is made by a kafka listener (response back from the createInvoice).

Sometimes the process executed by the serviceA fails. With the current logic, we are just storing the response, but the invoice is not created.

I’m looking to add two things:

  1. Add a retry on the createInvoice call (probably with a max_retry limit)
  2. Add the possibility of completing the awaitForKafkaResponseWorkflow method manually in case the task is completed (create the invoice directly in the third-party system).

The solution I’ve thought of is to:

  1. Add a loop to retry the createInvoice activity until the response is successful
  2. Add an endpoint that triggers the signal for completion.

Is this the best approach? Is this even possible?

For the possible solution I was thinking something like this

fun worklowMainMethod(): Any {
    val userResponse = createAndWaitForUserResponse() // child workflow

    while (successRespoonse && retryCount < max_retries) {
        createInvoice(userResponse) // activity that sends message to kafka
        val kafkaResponse = awaitForKafkaResponseWorkflow() // child workflow
    }
    if (!successResponse) {
        val kafkaResponse = awaitForKafkaResponseWorkflow() // child workflow
    }

    val response = sendEmail(kafkaResponse) // activity that sends email
}

Thank you!!

2
  1. Add a loop to retry the createInvoice activity until the response is successful

Activities are already retried according to their retry options. So you don’t need to change the workflow code to enable this activity retry. You can change the retry options if you don’t like the defaults.

  1. Triggering a signal from an endpoint is fine. What is the reason for using a child workflow to wait for the signal?
3

Activities are already retried according to their retry options. So you don’t need to change the workflow code to enable this activity retry. You can change the retry options if you don’t like the defaults.

Yes, the activity itself never fails because it’s just sending a message to kafka. What it “fails” is the whole createInvoice operation because, in the response back (awaitForKafkaResponseWorkflow), I get a 500 error (example) because the third party is down.
That’s why I was thinking of wrapping in a loop not only the call to kafka (createInvoice) but also the analysis of the response.

  1. Triggering a signal from an endpoint is fine. What is the reason for using a child workflow to wait for the signal?

I use a child workflow because I need to not only do the invoice creation but also the bill one. So I’m creating two workflows and adding a list of promises in the main workflows to wait for them.

Here is the code with more detail.

val createInvoiceAndBillActivity() // it sends a message to kafka. I expect two messages back (createInvoiceResponse and createBillResponse).
val promises = List<Promises>()
promises.add(awaitChild("invoice"), awaitChild("bill"))

promises.map { p -> promises.get() }

fun awaitChild(type) {
  return Async.function(childworkflow<...>())
}
  1. Using the loop to wrap the (activity + response), and depending on the answer to retry, is the best way to do it?
  2. Is there any other way to do the wait for the response of multiple messages?
4

You don’t have to use child workflows for more complex logic. You can always create classes that own that logic and run them inside the same workflow.

But in your case, I recommend moving the retry logic of the whole async operation (both createInvoice and wait for the reply signal) into a child workflow. The reason is that each retry would add events to the workflow history. So, the child can call continue-as-new after a few retries. The parent doesn’t see continue-as-new of the child as it waits for its completion. This way, you can retry indefinitely without growing the parent’s history.

You can create promises using Workflow.newPromise and complete them in the signal handlers. Another option is to use Workflow.await(()->signal1Received && signal2Received) condition.