Async execution of activities with guaranteed order outside of the current workflow

Community Support
,
1

We have the following use case which might be useful to implement in Temporal or Temporal SDKs (we use Java SDK):

Workflow executes a sequence of activities similar to this:

customerDbActivities.updateStatus(LOCKED);
kafkaActitivities.sendEvent(<customerId : 1, status : LOCKED>);

// execute some operations and update customer
//invoke some activities and child workflows

// update customer status once again
customerDbActivities.updateStatus(READY);
kafkaActitivities.sendEvent(<customerId : 1, status : READY>);

Kafka events are emitted for external consumers only. If Kafka is slow, unavailable or there are some network issues - these should not block the execution and completion of the main workflow, but the events should eventually be delivered to Kafka in the invoked order.
The way we implemented it is in kafkaActivities we do not actually interact with Kafka, instead we create a separate workflow (e.g. SendKafkaEventsWorkflow) using signalWithStart, and send the event to this workflow via a signal method.
The workflow just accumulates the events in the internal queue and executes KafkaProducer.send().get for each event one-by-one with infinite retries.
To avoid issues with history / memory size, the workflowId includes customerId, so each customer has its own SendKafkaEventsWorkflow workflow execution.

Would be nice if Temporal provided an out of the box way to execute / schedule activities like this: with user-provided partitioning key and with ordered execution outside of the parent workflow (the activities are guaranteed to be executed in the invoked order even after their parent workflow completes).

2 Likes
3

You are correct, Temporal doesn’t guarantee relative async activity exec order unless its ordered by your workflow code, more info here.

Thanks for the feature request, opened issue here.