Ordering of requests to downstream service in back pressure scenario

Hello Temporal team,

Here is the scenario:

Say we have 10 API requests resulting in 10 workflows calling downstream service A. Service A might become overloaded or experience an outage, causing Temporal activity retries to occur. Is there a way in Temporal to ensure that the first request’s activity retry gets serviced before the 2nd, 3rd, etc…?

I’'m trying to avoid a race condition where 2nd, 3rd, etc… gets priority and the first request continues to retry, which would lead to long processing time for the user.

I was thinking of using a Kafka queue between Temporal activity and the downstream service, but this may be redundant.

Thanks,
Richard

Temporal doesn’t provide any guarantee around activity execution ordering. So it cannot guarantee that activity retries will be delivered in some specific order.

To avoid overloading a downstream service Temporal supports task queue rate limiting. When configured the tasks are not dispatched to the workers faster than the limit.

I was thinking of using a Kafka queue between Temporal activity and the downstream service, but this may be redundant.

I doubt that this would improve quality and availability of your system.

1 Like