Is retry policy applicable to platform (temporal) error?

Community Support
1

My use case is to retry for specific error types and my understanding is that Using retry policy alongside wrapping my errors into retryable/non-retryable error would work.

My question is that: is retry policy applicable to platform (temporal) error? in other words, if an error is thrown by temporal itself, will the activity be re-executed? assume retry policy allows retry.

2

There are specific scenarios when an activity task can be lost in transit after it is marked as started. In this case, it will be marked as timed out after StartToCloseTimeout and retried according to the retry options.

3

@maxim

Is there a chance an error may be returned/thrown after user defined activity code has been executed?

My activity is calling an external API, which is not able to provide idempotency guaranteed, so I want my activity to retry only for some specific error types.

4

Return an ApplicationError with NonRetryable set to true to ignore the retry policy.

BTW Have you did this course?

Crafting an Error Handling Strategy | Learn Temporal

5

Thanks for pointing out.

Just realize that activity timeout will trigger retry for the activity, which may be dangerous for my use case, since the external API does not have idempotency in place.

Is there a way to disable retry for activity timeout and allow retry for specific error types (error type can be decided based on the external API response).

6

Currently there is no way to have a different retry options for different error types besides marking ApplicationError as non retryable.

You have to perform all retries from the workflow code to support your use case. To disable all server side retries set maxAttempts to 1.

7

Hey,

If an activity fails due to a temporary platform problem, like a network hiccup or server issue, and a retry is allowed, it will automatically run again. But if the problem is permanent, it won’t be retried.

8

Hey! Could you please clarify what are these specific scenarios? :smiling_face_with_tear: I guess my system encountered some of them and I’m trying to cope with them

9

For example worker crashing after receiving a task but before invoking the activity function. Or frontend crashing while delivering the task.

I’m trying to cope with them

Using timeouts and retry options is the right way to cope with them.

10

For example worker crashing after receiving a task but before invoking the activity function. Or frontend crashing while delivering the task.

Thanks, I agree that timeouts and retries are needed for reliability.

In my case, nothing crashed as far as I can see: the pod had no restarts, the app process stayed alive, and I verified that the worker was able to process other tasks.

I’ve seen similar cases with both Python and Java SDK workers.

So I’m trying to understand whether there are any less obvious scenarios where an Activity can be marked as STARTED / assigned to a worker, but never reach my Activity function code.

And more importantly: how would you recommend investigating such cases and reducing the probability of them happening?

11

Did the service had any restarts or shard movements at the time? Or networking issues?
CPU spikes can also lead to task loss.