Ordering of requests to downstream service in back pressure scenario

Community Support
1

Hello Temporal team,

Here is the scenario:

Say we have 10 API requests resulting in 10 workflows calling downstream service A. Service A might become overloaded or experience an outage, causing Temporal activity retries to occur. Is there a way in Temporal to ensure that the first request’s activity retry gets serviced before the 2nd, 3rd, etc…?

I’'m trying to avoid a race condition where 2nd, 3rd, etc… gets priority and the first request continues to retry, which would lead to long processing time for the user.

I was thinking of using a Kafka queue between Temporal activity and the downstream service, but this may be redundant.

Thanks,
Richard

2

Temporal doesn’t provide any guarantee around activity execution ordering. So it cannot guarantee that activity retries will be delivered in some specific order.

To avoid overloading a downstream service Temporal supports task queue rate limiting. When configured the tasks are not dispatched to the workers faster than the limit.

I was thinking of using a Kafka queue between Temporal activity and the downstream service, but this may be redundant.

I doubt that this would improve quality and availability of your system.

1 Like