Workflow Task Failed - Workflow not failed

jayk · September 14, 2022, 10:12pm

Hi,

I’m developing a workflow and I’ve done something wrong and I get the following error:

In my terminal logs, I can see this:

1f3dd74b7a **Attempt 73** CachedPreviousStartedEventID 6 TaskFirstEventID 1 TaskStartedEventID 6 PreviousStartedEventID 0
2022/09/14 23:07:46 DEBUG WfData:  Namespace default TaskQueue TestTQ WorkerID 25159@KJAY-M-MGGN@ WorkflowType ExecuteWorkflow WorkflowID test759d0db1-47ec-47a5-83da-5e4719f5ec3c RunID 611799b5-1fba-41d5-b0d9-e21f3dd74b7a Attempt 1 {"stringParam":"Jay"}
WFData: {"stringParam":"Jay"}
Operation State
Function Ref:  uppercase
2022/09/14 23:07:46 DEBUG ExecuteActivity Namespace default TaskQueue TestTQ WorkerID 25159@KJAY@ WorkflowType ExecuteWorkflow WorkflowID test759d0db1-47ec-47a5-83da-5e4719f5ec3c RunID 611799b5-1fba-41d5-b0d9-e21f3dd74b7a Attempt 1 ActivityID 8 ActivityType Uppercase
2022/09/14 23:07:46 INFO  Task processing failed with error Namespace default TaskQueue TestTQ WorkerID 25159@KJAY@ WorkerType WorkflowWorker Error BadScheduleActivityAttributes: A valid StartToClose or ScheduleToCloseTimeout is not set on command.

Now I understand why I’m getting this error, and it’s easy enough to fix. My question is, in this instance, why is the workflow status not failed? The workflow status is running and I have to terminate this manually.

Thanks,
Jay.

maxim · September 15, 2022, 9:31pm

This is by design. We don’t want workflows to fail on unexpected panics. This allows fixing the code without losing workflow instances. You can disable this behavior by setting WorkerOptions.WorkflowPanicPolicy to FailWorkflow.

tihomir · September 15, 2022, 9:37pm

Now I understand why I’m getting this error, and it’s easy enough to fix

Yes, just for reference to other readers, ActivityOptions must have a StartToClose and/or ScheduleToCloseTimeout set.

why is the workflow status not failed

The terminal log you have I believe is a server log. Means that your workflow worker sent the schedule activity invocation command to the server and server noticed that its missing timeout config. For command validation errors your workflow exec gets blocked and your workflow task is retried waiting on a fix.
You should be able to fix your workflow code and restart your workers and the execution would be able to continue on the next workflow task retry.

jayk · September 15, 2022, 10:15pm

Thanks @maxim and @tihomir.

Vishal_Thirumoorthy · March 13, 2023, 2:20pm

Hi @maxim and @tihomir,

I recently came across the same problem with my workflow where my WorkflowTask failed but my workflow did not. I went looking for the solution but i could find the panicPolicy being supported for GO. I am unable to find the same for java.

does Java support the panic policy, if yes am I searching it in the wrong place ?

Regards,
Vishal.

maxim · March 14, 2023, 12:36am

In Java, you can specify which exceptions lead to workflow failures. Use WorkflowImplmentationOptions.setFailWorkflowExceptionTypes.

Vishal_Thirumoorthy · March 14, 2023, 6:37am

Hi @maxim, I tried but still the issue seems to persist. Please refer screenshot below.

Also passing Throwable.class which should fail workflow on any exception does not fail the workflow.

github.com

temporalio/sdk-java/blob/master/temporal-sdk/src/main/java/io/temporal/worker/WorkflowImplementationOptions.java

/*
 * Copyright (C) 2022 Temporal Technologies, Inc. All Rights Reserved.
 *
 * Copyright (C) 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 *
 * Modifications copyright (C) 2017 Uber Technologies, Inc.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this material except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

This file has been truncated. show original

maxim · March 14, 2023, 3:50pm

It looks like this option is ignored when the failure is reported by the service. I filed a bug report.

Chandan_Bhattad · June 3, 2023, 12:43pm

is similar option available in python sdk?

Rami_Loiferman · August 9, 2023, 6:18am

IMHO this may cause bad user experience, for example in case a user clicked on a button in the browser and this triggered a workflow,
Now the user pooling this workflow to completion but it never completes,

Another question in case the workflow timeout is 5 minutes seems not real to be able to write a fix, code review and deployment

maxim · August 9, 2023, 1:07pm

For such use cases, set the option always to fail workflow on unexpected errors.

Failing workflows on an unexpected bug can be catastrophic for certain scenarios when returning an error is not an option.

Rami_Loiferman · August 9, 2023, 1:47pm

Thak you for the answer,
We are struggling to find a way to change this option in typescript SDK, Is there a way?
And if there a cluster-level configuration for that even more suitable for us