# Workflow stuck at \`WorkflowTaskScheduled\` due to \`Persistent store operation Failure\` on the Temporal History **URL:** \ **Category:** Server Deployment\ **Created:** [January 24, 2024, 7:06pm UTC](https://community.temporal.io/t/workflow-stuck-at-workflowtaskscheduled-due-to-persistent-store-operation-failure-on-the-temporal-history/10778 "2024-01-24T19:06:37Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![jwang97](https://avatars.discourse-cdn.com/v4/letter/j/f17d59/32.png) [@jwang97](https://community.temporal.io/u/jwang97)\ **Post date:** [January 24, 2024, 7:06pm UTC](https://community.temporal.io/t/workflow-stuck-at-workflowtaskscheduled-due-to-persistent-store-operation-failure-on-the-temporal-history/10778/1 "2024-01-24T19:06:37Z")
During periods of high load, we are seeing some workflows being stuck at `WorkflowTaskScheduled` for multiple days despite having active workers on the queues. Looking into our datadog logs for the workflow, we are seeing this error: ```auto Persistent store operation Failure ``` with the following trace: ```auto go.temporal.io/server/common/log.(*zapLogger).Error /home/builder/temporal/common/log/zap_logger.go:156 go.temporal.io/server/service/history/workflow.createWorkflowExecution /home/builder/temporal/service/history/workflow/transaction_impl.go:381 go.temporal.io/server/service/history/workflow.(*ContextImpl).CreateWorkflowExecution /home/builder/temporal/service/history/workflow/context.go:334 go.temporal.io/server/service/history/api/startworkflow.(*Starter).createBrandNew /home/builder/temporal/service/history/api/startworkflow/api.go:250 go.temporal.io/server/service/history/api/startworkflow.(*Starter).Invoke /home/builder/temporal/service/history/api/startworkflow/api.go:178 go.temporal.io/server/service/history.(*historyEngineImpl).StartWorkflowExecution /home/builder/temporal/service/history/history_engine.go:358 go.temporal.io/server/service/history.(*Handler).StartWorkflowExecution /home/builder/temporal/service/history/handler.go:588 go.temporal.io/server/api/historyservice/v1._HistoryService_StartWorkflowExecution_Handler.func1 /home/builder/temporal/api/historyservice/v1/service.pb.go:1157 go.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1 /home/builder/temporal/common/rpc/interceptor/retry.go:63 go.temporal.io/server/common/backoff.ThrottleRetryContext /home/builder/temporal/common/backoff/retry.go:145 go.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept /home/builder/temporal/common/rpc/interceptor/retry.go:67 google.golang.org/grpc.getChainUnaryHandler.func1 /go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195 go.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept /home/builder/temporal/common/rpc/interceptor/rate_limit.go:88 google.golang.org/grpc.getChainUnaryHandler.func1 /go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195 go.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept /home/builder/temporal/common/rpc/interceptor/telemetry.go:165 google.golang.org/grpc.getChainUnaryHandler.func1 /go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195 go.temporal.io/server/common/metrics.NewServerMetricsTrailerPropagatorInterceptor.func1 /home/builder/temporal/common/metrics/grpc.go:113 google.golang.org/grpc.getChainUnaryHandler.func1 /go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195 go.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1 /home/builder/temporal/common/metrics/grpc.go:66 google.golang.org/grpc.getChainUnaryHandler.func1 /go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195 go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1 /go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.42.0/interceptor.go:344 google.golang.org/grpc.getChainUnaryHandler.func1 /go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195 go.temporal.io/server/common/rpc.ServiceErrorInterceptor /home/builder/temporal/common/rpc/grpc.go:145 google.golang.org/grpc.chainUnaryInterceptors.func1 /go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1186 go.temporal.io/server/api/historyservice/v1._HistoryService_StartWorkflowExecution_Handler /home/builder/temporal/api/historyservice/v1/service.pb.go:1159 google.golang.org/grpc.(*Server).processUnaryRPC /go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1376 google.golang.org/grpc.(*Server).handleStream /go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1753 google.golang.org/grpc.(*Server).serveStreams.func1.1 /go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:998 ``` further relevant info: ```auto store-operation: create-wf-execution logging-call-at: transaction_impl.go:381 error: shard status unknown ``` temporal version: 1.22.2 Is this a bug with temporal? Any guidance on how we can try to prevent these errors in the future? ---
**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\ **Post date:** [January 28, 2024, 10:00am UTC](https://community.temporal.io/t/workflow-stuck-at-workflowtaskscheduled-due-to-persistent-store-operation-failure-on-the-temporal-history/10778/2 "2024-01-28T10:00:27Z")
> error: shard status unknown This typically means the shard is not operational as in service received timeout from the db, this call was retried a number of times resulting in db timeout continuously. Do you see any errors on db side? On server metrics side can you look at persistence latencies and failures (metrics `persistence_latency` and `persistence_error_with_type`) ---
**Author:** ![jwang97](https://avatars.discourse-cdn.com/v4/letter/j/f17d59/32.png) [@jwang97](https://community.temporal.io/u/jwang97)\ **Post date:** [January 30, 2024, 9:57pm UTC](https://community.temporal.io/t/workflow-stuck-at-workflowtaskscheduled-due-to-persistent-store-operation-failure-on-the-temporal-history/10778/3 "2024-01-30T21:57:59Z")
We do see errors on the db side. We have tuned some of our RPS limits to prevent the db from getting hammered. Outside of this, do you have any suggestions on how we can ensure that workflows get eventually persisted no matter how many retries are needed from the db?