Hey, I am Junaid,
I am playing around with temporal for some time, I am facing an issue. it would be really helpful if some could help with this.
while creating a workflow that has a blocking activity (running in an infinite loop with context cancel exit ), that activity doing some logic internally in the loop with the timer (in seconds, so I am not using cronjobs). I am experiencing multiple calls to the activity from the workflow and not able to cancel the activity.
workflow execution code provided below.
wfopts := client.StartWorkflowOptions{
TaskQueue: temporalTaskQueue,
ID: "CreateMatches_" + tournament.Id,
WorkflowIDReusePolicy: enums.WORKFLOW_ID_REUSE_POLICY_ALLOW_DUPLICATE_FAILED_ONLY,
WorkflowTaskTimeout: 1 * 24 * 7 * 52 * time.Hour,
RetryPolicy: &temporal.RetryPolicy{InitialInterval: time.Second},
}
wr, err := s.wfClient.ExecuteWorkflow(ctx, wfopts, workflowCreateMatches, mProfiles, houseId, tournament.Id)
err = wr.Get(ctx, &err)
and below is the activity invocation code within workflow “workflowCreateMatches”
func (ctx workflow.Context, mProfiles []MatchProfile, houseId, tourId string){
actx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
ScheduleToStartTimeout: 10 * time.Minute,
StartToCloseTimeout: 1 * 24 * 7 * 52 * time.Hour,
HeartbeatTimeout: 5 * time.Minute,
WaitForCancellation: false,
RetryPolicy: &temporal.RetryPolicy{InitialInterval: time.Minute},
})
future := workflow.ExecuteActivity(actx, new(DirectorSvc).CreateAndAssignMatches, mProfiles, houseId, tourId)
err := future.Get(ctx, nil)
if err != nil {
if temporal.IsCanceledError(err) {
return nil
}
}
and the activity implementation is looking like this.
func (d *DirectorSvc) CreateAndAssignMatches(ctx context.Context, mProfiles []MatchProfile, houseId, tourId string) error {
for {
select {
case <-ctx.Done():
return nil
case <-time.After(time.Duration(logicInterval) * time.Second):
{
if time.Now().After(heartBeatTime.Add(time.Minute)) {
activity.RecordHeartbeat(ctx, "")
log.Debug(ctx, "HeartBeat", log.KV("tourId", tourId))
heartBeatTime = time.Now()
}
//logic
}
}
}
return nil
}
I am trying to call this workflow but facing two issues that not able to solve
-
the activity “CreateAndAssignMatches” getting called multiple times … like initially, log " HeartBeat" is coming was coming in 1 min interval… then it getting increased and it was coming like 6 times in a minute, so as per log, I am sure that this activity getting called multiple times, within this single workflow.
-
I am trying to cancel this workflow so that I can exit from the activity by context cancel (i want to somehow stop the loop).
err := s.wfClient.CancelWorkflow(ctx, "CreateMatches_" + tournament.Id,, "") if err != nil { }
but it says it canceled the workflow, but actually the activity keeps on running (i think context is not canceling, even though I am recording the heartbeat.
my doubts are :
- why this single activity getting called multiple times and running separately on the worker ( how multiple invocations happened to the activity, I am sure it is not coming from retry. because there was no failure happening ).
- how can I exit from this blocking activity. (using record activity in go)
the steps is shown in temporal UI are,
1 WorkflowExecutionStarted. workflowType.name. workflowCreateMatches
2 WorkflowTaskScheduled taskQueue.name matchCreate_tq
taskQueue.kind Normal
startToCloseTimeout 60.
3. WorkflowTaskStarted
4 WorkflowTaskCompleted
5 ActivityTaskScheduled activityId 5
activityType.name CreateAndAssignMatches
after exactly 1hr and 45s later
6. ActivityTaskTimedOut 1h 45s (+1h 45s)
failure.timeoutFailureInfo.timeoutType ScheduleToStart
failure.failureInfo. timeoutFailureInfo
scheduledEventId. 5
7 WorkflowTaskScheduled 1h 45s
taskQueue.name 8255f05a42c0:51047173-1314-40e7-82c5-1888c216503a
taskQueue.kind Sticky
startToCloseTimeout 60
8. WorkflowTaskTimedOut. 1h 50s (+5s)
scheduledEventId 7
startedEventId 0
timeoutType ScheduleToStart
9 WorkflowTaskScheduled 1h 50s
taskQueue.name matchCreate_tq
taskQueue.kind Normal
startToCloseTimeout 60
10. WorkflowTaskStarted
11. WorkflowTaskCompleted
12 WorkflowExecutionFailed 1h 50s
failure.message: failed to exicute CreateMatches: activity task error (scheduledEventID: 5,
TimeoutType: ScheduleToStart, Cause: <nil>
Failure.source GoSDK
failure.cause.message:
activity task error (scheduledEventID: 5, startedEventID: 0, identity: ): TimeoutType:
ScheduleToStart, Cause: <nil>
Failure.cause.source GoSDK
the entire program is stopping after this 1hr 50s, i am not able to understand what is going on.
please ignore the parameters
Please help with this.
Thanks in advance,
Junaid