Is it possible to catch exceptions thrown by activities inside signal method

[gaurav_21]

Hello, we are running an activity inside a signal method. When the activity fails, it throws an exception. Is it possible to catch exceptions thrown by activities inside signal method. Because currently the workflow just throws an exception of type InternalWorkflowTaskException and is stuck there until the workflow times out

io.temporal.internal.statemachines.InternalWorkflowTaskException: Failure handling event 36 of type 'EVENT_TYPE_WORKFLOW_TASK_STARTED' during execution. {PreviousStartedEventId=30, WorkflowTaskStartedEventId=36, CurrentStartedEventId=36} 
io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:263)
io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:242)
io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:216)
io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:190)
io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:137)
io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:129)
io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:98)
io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handleTask(WorkflowWorker.java:293)
io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:237)
io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:178)
io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:834)
type or paste code here

[maxim]

Did you try a try-catch block around activity invocation?

[gaurav_21]

Yes.
Actually I am getting this -

{
"level": "error",
  "ts": "2023-03-25T19:15:11.644Z",
  "msg": "encounter error when describing the mutable state",
  "service": "worker",
  "error": "context deadline exceeded",
  "wf-namespace-id": "bff0fb3b-316b-4ce8-a2b2-d5a0cb255222",
  "wf-id": "ACC425357_NOMINEE_DATA_COLLECTION",
  "wf-run-id": "a87525cc-0dcb-4405-a038-ba6325e99160",
  "wf-tree-id": "a87525cc-0dcb-4405-a038-ba6325e99160",
  "wf-branch-id": "7db36705-b4ed-43bc-9d5d-90f748c93d1f",
  "logging-call-at": "scavenger.go:267",
  "stacktrace": "go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/worker/scanner/history.(*Scavenger).handleTask\n\t/home/builder/temporal/service/worker/scanner/history/scavenger.go:267\ngo.temporal.io/server/service/worker/scanner/history.(*Scavenger).taskWorker\n\t/home/builder/temporal/service/worker/scanner/history/scavenger.go:199"
}

Not sure what exactly is the issue.

[gaurav_21]

@gaurav_21

encounter error when describing the mutable state

What does this error mean?

[tihomir]

"msg": "encounter error when describing the mutable state",
 "service": "worker",
 "error": "context deadline exceeded",

Scavenger workflow that runs on the worker service and tries to garbage collect event histories that for some reason did not get cleaned up on namespace retention checks if mutable state (in history shard) still exists for and execution and times out (deadline exceeded).

Is this error transient? If so think you can ignore.

[gaurav_21]

Unfortunately, not.
This error is leading to some of our users being stuck and not move ahead in their journey.

[tihomir]

Any errors in history service logs?
If you have server metrics enabled try:

sum(rate(service_error_with_type{service_type="history"}[5m])) by (error_type)

(same for frontend service errors, service_type=“frontend”)

[gaurav_21]

[manasjain_groww]

Attaching graph for frontend service