Hi, I use temporal to execute workflow for application of configuration to external systems. External systems broadcasts result by message broker, and my consumer send signal of response to waiting workflow by WorkflowClient.SignalWorkflow and on another side workflow receives the response by channel ReceiveChannel.ReceiveWithTimeout. Everything works fine, but yesterday I’ve got panic panic [TMPRL1100] lookup failed for scheduledEventID to activityID.
I’ve read that it happens when non-deterministic error happens, but can’t understand where it come from.
Maybe you can help me to understand
Here the part of code of wf that receives signals
updateChan := workflow.GetSignalChannel(ctx, kafka.GetSignalName(cfg.ResponseTopic, request.ID))
for {
var event utils.Response
ok, _ := updateChan.ReceiveWithTimeout(ctx, cfg.ResponseTimeout, &event)
if !ok {
return fmt.Errorf("timeout waiting for update events from %s for requestID: %s", cfg.ExternalSystem, request.ID)
}
if event.Status == string(configurationModels.ConfiguratorStatusFailed) {
return fmt.Errorf("got error from %s: requestID=%s, error=%s", cfg.ExternalSystem, event.RequestID, event.Error)
}
if event.Status == string(configurationModels.ConfiguratorStatusAccepted) {
continue
}
if event.Status == string(configurationModels.ConfiguratorStatusInProgress) {
if err := activities.PublishKafkaMessage(ctx, request, configurationModels.ConfigurationStatusInProgress, "", updateTopic); err != nil {
return err
}
continue
}
if event.Status == string(configurationModels.ConfiguratorStatusCompleted) {
break
}
logger.Errorf("unknown status from %s: requestID=%s, status=%s", cfg.ExternalSystem, event.RequestID, event.Status)
}
The trace of panic
{
"message": "[TMPRL1100] lookup failed for scheduledEventID to activityID: scheduleEventID: 61, activityID: 61",
"source": "GoSDK",
[panic]:\ngo.temporal.io/sdk/internal.panicIllegalState(...)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.27.0/internal/internal_command_state_machine.go:440\ngo.temporal.io/sdk/internal.(*commandsHelper).handleActivityTaskScheduled(0xc000d6b240, {0xc0005ef5d8, 0x2}, 0x5?)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.27.0/internal/internal_command_state_machine.go:998 +0xfb\ngo.temporal.io/sdk/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc000a4d200, 0xc0002d9110, 0x3?, 0x0)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.27.0/internal/internal_event_handlers.go:1152 +0x290\ngo.temporal.io/sdk/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc001080b40, 0xc000d6e1e0)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.27.0/internal/internal_task_handlers.go:1149 +0x193d\ngo.temporal.io/sdk/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc0002c4680, 0xc000d6e1e0, 0xc001080b40, 0xc000d6eb10)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.27.0/internal/internal_task_handlers.go:914 +0x3c5\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).processWorkflowTask(0xc00057d560, 0xc000d6e1e0)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.27.0/internal/internal_task_pollers.go:357 +0x3a3\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).ProcessTask(0xc00057d560, {0x1237920?, 0xc000d6e1e0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.27.0/internal/internal_task_pollers.go:321 +0x87\ngo.temporal.io/sdk/internal.(*baseWorker).processTask(0xc000668420, {0x12384e0?, 0xc000d5c850})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.27.0/internal/internal_worker_base.go:518 +0x15b\ngo.temporal.io/sdk/internal.(*baseWorker).processTaskAsync.func1()\n\t/go/pkg/mod/go.temporal.io/sdk@v1.27.0/internal/internal_worker_base.go:369 +0x45\ncreated by go.temporal.io/sdk/internal.(*baseWorker).processTaskAsync in goroutine 132\n\t/go/pkg/mod/go.temporal.io/sdk@v1.27.0/internal/internal_worker_base.go:365 +0xb8",
"encodedAttributes": null,
"cause": null,
"applicationFailureInfo": {
"type": "PanicError",
"nonRetryable": true,
"details": null
}
}
Here the history of execution of failed workflow
Server version: v1.20.3
SDK version: v1.27.0