we are running temporal for couple of years now and things were completely fine.
there is a org mandate to do mtls across all pods and we now introduced istio side car.
after rolling out istio changes we start noticing
History is out of order error like below
,“level”:“ERROR”,“stack_trace”:"java.lang.RuntimeException: Failure processing workflow task. WorkflowId=env2230208, RunId=8f950fa4-d5bb-4df8-8d61-8274b12dbd2f, Attempt=28\n\tat io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:467)\n\tat io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:307)\n\tat io.temporal.internal.worker.PollTaskExecutor.lambda$process$1(PollTaskExecutor.java:101)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: java.lang.IllegalStateException: History is out of order. There is a gap between the last event workflow state machine observed and currently handling event. Last processed eventId: 323, handling eventId: 328\n\tat com.google.common.base.Preconditions.checkState(Preconditions.java:759)\n\tat io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:286)\n\tat io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.applyServerHistory(ReplayWorkflowRunTaskHandler.java:257)\n\tat io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:239)\n\tat io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:165)\n\tat io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:135)\n\tat io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:100)\n\tat io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handleTask(WorkflowWorker.java:476)\n\tat io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:367)\n\tat io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:307)\n\tat
is the temporal db corrupted?
is there any specific recommendation on how to roll out istio for temporal workflods?