Workflow timedout due to activity BLOCKED on Feature.get

Community Support
,
1

hi,

We are observing random workflow timeouts and we have the following stack trace.

workflow-method-30DBBBA2B0B145C0BF6BB3EAC73EC1CC@AXkBAgA-TR-ggwc--89671688-e671-4461-ba60-7251ed17f99f: (BLOCKED on Feature.get)
io.temporal.internal.sync.CompletablePromiseImpl.getImpl(CompletablePromiseImpl.java:84)
io.temporal.internal.sync.CompletablePromiseImpl.get(CompletablePromiseImpl.java:74)
com.walmart.itemingestion.util.ItemWorkflowUtil.executeDependentSteps(ItemWorkflowUtil.java:125)
com.walmart.itemingestion.util.ItemWorkflowUtil.executeRoutes(ItemWorkflowUtil.java:46)
com.walmart.itemingestion.workflow.ItemSetupV2WorkflowImpl.executeV2(ItemSetupV2WorkflowImpl.java:118)
sun.reflect.GeneratedMethodAccessor732.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:317)
io.temporal.common.interceptors.WorkflowInboundCallsInterceptorBase.execute(WorkflowInboundCallsInterceptorBase.java:37)
com.walmart.itemingestion.interceptor.WorkflowInterceptor.execute(WorkflowInterceptor.java:23)

Additional info
SDK: Java Sdk 1.7.1
Server: 1.9.2
Activities: Local activities
Exported the history to a json file - seems to be of 1.2MB size. How do I share this file?

2

First look seems you have local activity that is never completing. Worth getting thread dump
of the process and take a look to confirm this.

Another thing to look at is if your local activity keeps retrying forever, there is an open issue here regarding local activity startToClose timeout as well as this one regarding completion callback issues.

Could you show parts of your code where you invoke local activities? Please show local activity config as well.

Exported the history to a json file - seems to be of 1.2MB size. How do I share this file?

could you dm it to me by chance?

3

Tagging @spikhalskiy

4

Hey @ravikiran,

Please reach out in Temporal Slack to me (@Dmitry) and let’s investigate together. There are several reports of this issue randomly appearing with Local Activities in the last year where people usually don’t follow-up with information and I would love to work with you to dig it out and make it fixed.

5

The local activity is performing a http request with a timeout configured. So we get a timeout exception and cannot enter a “never completing” state. I do not see any exceptions so retry is not seen.

1 Like
6

For those that aren’t in Slack, this issue has been fixed in Java SDK v1.14.0

2 Likes