That is the requirement Maxim, we have multiple third party systems that can be down or fail.
There we want to manually restart the workflow from failed point.
Have you considered just retrying activities indefinitely while these third party systems are down?
This way you donât need manual restarts at all.
for some recoverable error , we can go for retries but if failure happens due to bad data , this will keep failing forever.
the user base which we are targeting is in millions by next year, so not sure how efficient retries would be.
Also in some of our use cases we are using Kafka for getting the events.
for some recoverable error , we can go for retries but if failure happens due to bad data , this will keep failing forever.
I see. In this case, fixing the workflow code and reset is the right way to go.
the user base which we are targeting is in millions by next year, so not sure how efficient retries would be.
This shouldnât be a problem as you can rate control activity executions.
Also in some of our use cases we are using Kafka for getting the events.
So in this case do you reset workflow to the state before getting the event?
I tried resetting like this:
ResetWorkflowExecutionRequest request = ResetWorkflowExecutionRequest.newBuilder().setNamespace(namespace)
.setWorkflowExecution(WorkflowExecution.newBuilder().setWorkflowId(workflowId).build())
.setWorkflowTaskFinishEventId(tid)
.setReason(âresetâ)
.build();
where in I was looking for last task id from temporal -dashboard and then passing it, how can i get this task finish event id programmatically ?
Also,
say my activity method is :
void hello(){
System.out.println("test 3");
throw new RuntimeException("test failed");
}
even when I retsart this activity , the print statement doesnât execute, activities are stateless so the entire activity method would be executed , right?
where in I was looking for last task id from temporal -dashboard and then passing it, how can i get this task finish event id programmatically ?
At this point, the best approach is to load the workflow execution history using GetWorkflowExecution gRPC API and find the last WorkflowTaskStarted event before the activity failure.
even when I retsart this activity , the print statement doesnât execute, activities are stateless so the entire activity method would be executed , right?
This is weird. The whole activity should be reexecuted.
is it not part of java sdk?
WorkflowServiceStubs service = WorkflowServiceStubs.newInstance();
GetWorkflowExecutionHistoryResponse result =
service
.blockingStub()
.getWorkflowExecutionHistory(
GetWorkflowExecutionHistoryRequest.newBuilder()
.setNamespace(namespace)
.setExecution(WorkflowExecution.newBuilder().setWorkflowId(workflowId).build())
.build());
Note that history can contain multiple pages. So you might need to call this API a few times using a page token.
Thanks a lot @maxim , it is really helpful
Hi @maxim,
I logged all the events from history , here is the list:-
which Id should be set in .setWorkflowTaskFinishEventId( tid ) to restart from failed activity ?
Events :: id 1 type : EVENT_TYPE_WORKFLOW_EXECUTION_STARTED
Events :: id 2 type : EVENT_TYPE_WORKFLOW_TASK_SCHEDULED
Events :: id 3 type : EVENT_TYPE_WORKFLOW_TASK_STARTED
Events :: id 4 type : EVENT_TYPE_WORKFLOW_TASK_COMPLETED
Events :: id 5 type : EVENT_TYPE_ACTIVITY_TASK_SCHEDULED
Events :: id 6 type : EVENT_TYPE_ACTIVITY_TASK_STARTED
Events :: id 7 type : EVENT_TYPE_ACTIVITY_TASK_FAILED
Events :: id 8 type : EVENT_TYPE_WORKFLOW_TASK_SCHEDULED
Events :: id 9 type : EVENT_TYPE_WORKFLOW_TASK_STARTED
Events :: id 10 type : EVENT_TYPE_WORKFLOW_TASK_COMPLETED
Events :: id 11 type : EVENT_TYPE_WORKFLOW_EXECUTION_FAILED
Events :: id 3 type : EVENT
Thanks @maxim , tried this way and it worked
for (HistoryEvent e : history.getEventsList()){
//TODO : paginated history events list
if(e.getEventType() == EventType.EVENT_TYPE_ACTIVITY_TASK_FAILED)
break;
if(e.getEventType() == EventType.EVENT_TYPE_WORKFLOW_TASK_STARTED)
tid = e.getEventId();
}
is it the correct way?
I think it should work
are you able to resume the same workflow from failed point again and again?
For me I can do it just once.
Code i am referring is
GetWorkflowExecutionHistoryResponse result =
service
.blockingStub()
.getWorkflowExecutionHistory(
GetWorkflowExecutionHistoryRequest.newBuilder()
.setNamespace(namespace)
.setExecution(WorkflowExecution.newBuilder().setWorkflowId(workflowId).build())
.build());
Do you mean the issue about resetting more than once?
If so yes, you need to set an unique request id with each ResetWorkflowExecution request, for example:
ResetWorkflowExecutionRequest resetWorkflowExecutionRequest =
ResetWorkflowExecutionRequest.newBuilder()
.setRequestId(UUID.randomUUID().toString())
.setNamespace("default")
.setWorkflowExecution(existingUntyped.getExecution())
.setWorkflowTaskFinishEventId(taskFinishEventId)
.build();
Thanks @tihomir , when I added requestId ,it solved my issue . I could see a new workflow instance.
- How do we get existingUntyped.getexecution ()?
- while restting some of the worlflows are getting timedout with cause - WORKFLOW_TASK_FAILED_CAUSE_RESET_WORKFLOW
- How do we get existingUntyped.getexecution ()?
You can get the untyped workflow stub:
- to an existing wokflow exec:
WorkflowStub existingUntyped = client.newUntypedWorkflowStub(workflowId, Optional.of(runId), Optional.of(workflowType));
- From a typed stub:
MyWorkflow workflow = client.newWorkflowStub(
MyWorkflow.class,
WorkflowOptions.newBuilder()
.setWorkflowId(workflowId)
.setTaskQueue(taskQueue)
.build()
);
// ...
WorkflowStub untypedWorkflowStub = WorkflowStub.fromTyped(workflow);
- while restting some of the worlflows are getting timedout with cause - WORKFLOW_TASK_FAILED_CAUSE_RESET_WORKFLOW
I think all of them should have this after reset. Reset seems to time out the wft and continue execution from it again.
Thanks @tihomir for your prompt response.
Hi, Iâm now trying to use Temporal but got an error when using the method: resetWorkflowExecution
io.temporal.failure.ActivityFailure: scheduledEventId=35, startedEventId=36, activityType='WaitJenkins', activityId='3fea59fa-d703-39d5-8612-4491a62f2ac7', identity='1@zcp-console-dev-main-va1-meta-5f4f5fb684-q7cqs', retryState=RETRY_STATE_NON_RETRYABLE_FAILURE
Here is my code:
WorkflowServiceStubs svc = WorkflowServiceStubs.newConnectedServiceStubs(wfsOpts, Duration.ofSeconds(5));
ResetWorkflowExecutionRequest request = xxx;
temporalSvc.blockingStub().resetWorkflowExecution(request);
My Agenda:
Users can retry the failed workflow manually. Also, they can choose to pick up where they failed or start over.
Thanks!
Hi @Haiping_Yu just referencing same question here.