Hi ,
Is there a metric whic i can track to identify bad deployments breaking some long running workflow.
.e.g when i introduce a change which is not backward compatible, i see that the workflow cannot make progress as the determinstiic nature is broken.
and i see error like
i.t.internal.worker.PollerOptions : uncaught exception
java.lang.RuntimeException: Failure processing workflow task. WorkflowId=mybroken-workflow-1, RunId=979ceb3a-aae7-445a-920b-04905ed1804a, Attempt=28904
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:349) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:279) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:79) ~[temporal-sdk-1.0.6.jar!/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_261]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_261]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_261]
Caused by: io.temporal.internal.replay.InternalWorkflowTaskException: Failure handling event 91 of ‘EVENT_TYPE_ACTIVITY_TASK_SCHEDULED’ type. IsReplaying=true, PreviousStartedEventId=89, workflowTaskStartedEventId=802, Currently Processing StartedEventId=89
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:193) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleEvent(ReplayWorkflowRunTaskHandler.java:140) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:180) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:150) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithEmbeddedQuery(ReplayWorkflowTaskHandler.java:201) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:114) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:319) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:279) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:73) ~[temporal-sdk-1.0.6.jar!/:na]
… 3 common frames omitted
Caused by: java.lang.IllegalStateException: COMMAND_TYPE_SCHEDULE_ACTIVITY_TASK doesn’t match EVENT_TYPE_ACTIVITY_TASK_SCHEDULED with EventId=91
at io.temporal.internal.statemachines.WorkflowStateMachines.assertMatch(WorkflowStateMachines.java:784) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.statemachines.WorkflowStateMachines.validateCommand(WorkflowStateMachines.java:737) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleCommandEvent(WorkflowStateMachines.java:272) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventImpl(WorkflowStateMachines.java:199) ~[temporal-sdk-1.0.6.jar!/:na]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:178) ~[temporal-sdk-1.0.6.jar!/:na]
… 11 common frames omitted
Is there a metric which can help me troubeshoot this , i tried temporal_activity_execution_failed but ti think that count does not increase as the activity itself is not invoked.
should i set an alarm on temporal_request_failure, with tags namespace=“temporal_system”,operation=“SignalWithStartWorkflowExecution” to identify a bad deployment?