Hey, I accidentally set the activity to retry forever, and then I need to set for it a maximum time of retries. I find Temporal stored the retry policy in the event, and even though I updated my workflow with the new retry policy, the old workflow executions and activities are still retried forever. What should I do for the history data? should I manually reset-as-new for all of them?
set the activity to retry forever, and then I need to set for it a maximum time of retries
If you want to limit activity retries recommended way is to use activity options ScheduleToClose timeout rather than RetryPolicy MaximumAttempts.
the old workflow executions and activities are still retried forever
You can fix your activity code and restart workers. The activity should then complete on the next retry.
Even tho you can change ActivityOptions in your workflow code without casing workflow determinism issues, I don’t think the updates are applied to pending activities (while retrying), retry options are recorded in the ActivityTaskScheduled event.
should I manually reset
You could reset your workflow to the WorkflowTaskStarted event right before the ActivityTaskScheduled event for this activity, in this case your running workflow execution that you are resetting will get terminated and reset is going to create a new execution with same workflow id, different run id, replay your history up to the reset point and then continue execution. This should then pick up your updated activity options. Imo fixing the activity failure in activity code would be a better approach.