Downstream client patterns with Temporal

Community Support
,
4

Yes, it is an anti-pattern if you own the downstream services.

Obviously, if you are calling into services that are part of another company or organization then you have no choice and you’ll have to forgo all the benefits outlined.

Here are some reasons to implement downstream services as activities directly:

Flow Control

  • If an activity worker is down it is not consuming activity tasks from the associated task queue. So no load on the service is generated and no error logs are produced.
  • Activity worker allows specifying per instance rate limit.
  • Activity worker allows specifying per instance limit on the number of parallelly executing activities.
  • Activity worker allows specifying per task list rate limit which is enforced by the service across any number of workers.
  • If there is a request spike and activities are requested faster than workers can (or allowed due to configured rate limits) then requests are backlogged in a task queue and processed later as soon as workers get spare capacity.

Compare it to the proposed downstream gRPC service approach:

  • If gRPC service is down activities are still executed and make requests to the service, possibly killing it with its requests.
  • If gRPC service is overloaded it has no way to push back on the request rate.
  • Additional load on Temporal service and activity workers to cycle for the failing activities.
  • No support for absorbing traffic spikes without overloading downstream service.

Routing and Load Balancing

  • You have to maintain completely separate routing and load balancing layer for RPC
  • This layer is not needed (besides the ability for Temporal workers to find Temporal Frontends) when using temporal activities directly.
  • Temporal supports routing requests to specific workers when needed. It can be achieved through RPC as well but might be nontrivial.

Long-Running Operations

  • RPC services don’t support long-running operations directly.
  • Temporal activities can have unlimited duration
  • Temporal activities support heartbeating to support fast worker failure detection
3 Likes