Current best approach for reporting the status of an async Workflow to an external service?

Community Support
1

What is the current best approach for reporting the status of a long-running async Workflow to an external service?

The external service is a planning tool that allows users to create both Temporal Workflows and non-Temporal “Workflows” that need to be manually completed by the users themselves. I was unable to find documentation for what I’m trying to achieve but perhaps I have just missed it.

Below is my current approach for starting the long-running async Temporal Workflows:

  1. A user uses the UI of the external service to send a POST request with Workflow related data to an API that executes Workflows. There are multiple endpoints; each one executes a different Workflow. The external service also creates a loosely matching entry in its DB to display the Workflow in its UI and to later recognize it should Temporal report its status.
  2. An asynchronous Workflow is started. It might sleep for months (the sleep is used as a schedule; since its a one-off using Schedules or cron doesn’t make sense) and may take hours to complete once woken up.

The most important statuses to report are whether the Workflow has completed successfully or failed. If possible, it would be nice to report that the Workflow execution has started (the after the sleep part).

One idea on how to report the status of a Workflow:

  • The Workflow has an infinitely retryable Activity that is executed after the initial sleep. This Activity would report that the “real” execution of the Workflow has started by sending a POST request to the API of the external service.
  • The Workflow has infinitely retryable Activities that are executed when any of the regular Activities fail.
    These Activities would report that the Workflow has failed by again sending a POST request to the API of the external service. After these activities, the Workflow would end by returning the relevant error.
  • The Workflow also has an infinitely retryable final Activity that reports the success of the Workflow. The logic is similar.
  • A side question, would there be any advantage to using Kafka where the API that executes Workflows would be the producer and the API of the external service would be the consumer?

Please let me know if there is a more optimal solution!

2

Your approach is fine. if you control the service that receives notifications, you can run the notification activity inside that service instead of exposing HTTP API. This way, no retries would be necessary in the majority of cases, as the service is not going to poll the activity task if it is not running.