Non retryable activity

Community Support
1

Hi. I have a case when i get data from third-party service and want to cancel activity and workflow with err.
I was trying to return nonretryable error.

func (v *VMOps) DeleteProtectionPlanActivity(ctx context.Context, data ProtectionPlanData) error {
        // .......
	if data.Ref == "" {
		return temporal.NewNonRetryableApplicationError("someaaa", "NewNonRetryableApplicationError", errors.New("failed"), "details here")
	}
      // .......
}
func workflow() {
// somewhere at workflow, also error type is added
	ctx = workflow.WithActivityOptions(
		ctx,
		workflow.ActivityOptions{
			StartToCloseTimeout: time.Hour,
			RetryPolicy: &temporal.RetryPolicy{
				NonRetryableErrorTypes: []string{"NewNonRetryableApplicationError"},
			},
		},
	)
	slog.Debug("started protection plan removing")
	err := workflow.ExecuteActivity(ctx, nullVMOps.DeleteProtectionPlanActivity, data).Get(ctx, &data.Ref)
	if err != nil {
		return err
	}
}

But it won’t help. Activity retries over and over again.

3:14PM DBG started protection plan removing service=unknown
3:14PM DBG ExecuteActivity ActivityID=5 ActivityType=DeleteProtectionPlanActivity Attempt=1 Namespace=default RunID=72d74bb1-15af-4422-816b-5ea5bc8f9eac TaskQueue=edgeclient WorkerID=46116@sc-mac-01014@ WorkflowID=DeleteProtectionPlan:(name01) WorkflowType=DeleteProtectionPlanWorkflow service=unknown
3:14PM ERR Activity error. ActivityType=DeleteProtectionPlanActivity Attempt=1 Error="someaaa (type: NewNonRetryableApplicationError, retryable: false): failed" Namespace=default RunID=72d74bb1-15af-4422-816b-5ea5bc8f9eac TaskQueue=edgeclient WorkerID=46116@sc-mac-01014@ WorkflowID=DeleteProtectionPlan:(name01) service=unknown
3:14PM DBG started protection plan removing service=unknown
3:14PM DBG ExecuteActivity ActivityID=5 ActivityType=DeleteProtectionPlanActivity Attempt=2 Namespace=default RunID=cebaa993-2bd5-4e3e-8b58-c99cf1aafcb9 TaskQueue=edgeclient WorkerID=46116@sc-mac-01014@ WorkflowID=DeleteProtectionPlan:(name01) WorkflowType=DeleteProtectionPlanWorkflow service=unknown
3:14PM ERR Activity error. ActivityType=DeleteProtectionPlanActivity Attempt=1 Error="someaaa (type: NewNonRetryableApplicationError, retryable: false): failed" Namespace=default RunID=cebaa993-2bd5-4e3e-8b58-c99cf1aafcb9 TaskQueue=edgeclient WorkerID=46116@sc-mac-01014@ WorkflowID=DeleteProtectionPlan:(name01) service=unknown
3:14PM DBG started protection plan removing service=unknown
3:14PM DBG ExecuteActivity ActivityID=5 ActivityType=DeleteProtectionPlanActivity Attempt=3 Namespace=default RunID=c6760fbe-d92d-4aff-9376-8c0116d7f589 TaskQueue=edgeclient WorkerID=46116@sc-mac-01014@ WorkflowID=DeleteProtectionPlan:(name01) WorkflowType=DeleteProtectionPlanWorkflow service=unknown
3:14PM ERR Activity error. ActivityType=DeleteProtectionPlanActivity Attempt=1 Error="someaaa (type: NewNonRetryableApplicationError, retryable: false): failed" Namespace=default RunID=c6760fbe-d92d-4aff-9376-8c0116d7f589 TaskQueue=edgeclient WorkerID=46116@sc-mac-01014@ WorkflowID=DeleteProtectionPlan:(name01) service=unknown

I see in logs that retryable: false, but workflow rerun forever. How to exit from workflow?

2

You don’t need to modify retry policy for non retryable activity. I cannot reproduce the problem on my laptop. How do you run the workflow? Are you using the unit testing framework?

3

sorry, i wrote it incorrect. Activity exits, but workflow retried according retry policy. Im not using unit testing framework. The goal is to break workflow completely without retries.
Here again example:

type NonRetryableError struct {
}

func (e *NonRetryableError) Error() string {
	return "some error from NonRetryableError"
}

func (v *VMOps) DeleteProtectionPlanActivity(ctx context.Context, data ProtectionPlanData) error {
	return temporal.NewNonRetryableApplicationError("someaaa", "NonRetryableError", &NonRetryableError{}, "details here")
}
// workflow started with this params
	options := temporalclient.StartWorkflowOptions{
		ID:                       workflowID,
		WorkflowExecutionTimeout: time.Hour,
		TaskQueue:                edgeclient.TaskQueue,
		RetryPolicy: &temporal.RetryPolicy{
			MaximumAttempts:        5,
			NonRetryableErrorTypes: []string{"NonRetryableError"},
		},
		Memo: memo,
	}

// workflow code
func DeleteProtectionPlanWorkflow(ctx workflow.Context, data *ProtectionPlanData) error {
	ctx = workflow.WithActivityOptions(
		ctx,
		workflow.ActivityOptions{
			StartToCloseTimeout: time.Hour,
			RetryPolicy: &temporal.RetryPolicy{
				NonRetryableErrorTypes: []string{"NonRetryableError"},
			},
		},
	)
	slog.Debug("started protection plan removing")
	err := workflow.ExecuteActivity(ctx, nullVMOps.DeleteProtectionPlanActivity, data).Get(ctx, nil)
	if err != nil {
		spew.Dump(err, checkRetryable(err))
		return err
	}
}

Workflow retries 5 times as written in policy. The problem is i don’t undestand how to propogate nonretry error from activity to workflow. This is dump, seems like error wrapped:

(*internal.ActivityError)(0x140003453b0)(activity error (type: DeleteProtectionPlanActivity, scheduledEventID: 5, startedEventID: 6, identity: 75369@sc-mac-01014@):
someaaa (type: NonRetryableError, retryable: false): some error from NonRetryableError (type: NonRetryableError, retryable: true))

So the only way i make it stop is workaround:

func checkRetryable(err error) error {
	var appErr *temporal.ApplicationError
	if errors.As(err, &appErr) && appErr.NonRetryable() {
		slog.Error("non retryable err found", "err", err)
		return temporal.NewNonRetryableApplicationError(appErr.Error(), "ApplicationError", err)
	}

	return err
}

Which looks a little bit messy. Is it the only way to propogate such errors to workflow?

4

Why are you retrying workflows? This is an anti pattern.

5

well, it’s not my code originally, so… by default anyway server allows workflow to be retryable. if this is an antipattern, will it be deprecated soon?

6

By default, workflows don’t have retry options and, thus, are not retried. So remove retry options passed to the StartWorkflow call and your code should work without any additional error handling.

7

okay, thnx!