[Error in UI]13 INTERNAL: ListClosedWorkflowExecutions failed. Error: elastic: Error 503 (Service Unavailable): all shards failed

Community Support
, ,
1

image
image3360×850 131 KB

13 INTERNAL: ListClosedWorkflowExecutions failed. Error: elastic: Error 503 (Service Unavailable): all shards failed [type=search_phase_execution_exception]

temporal frontend logs :: {"level":"error","ts":"2021-06-10T19:44:27.710Z","msg":"Operation failed with internal error.","service":"frontend","metric-scope":240,"error":"ListClosedWorkflowExecutions failed. Error: elastic: Error 503 (Service Unavailable): all shards failed [type=search_phase_execution_exception]","logging-call-at":"esVisibilityMetricClients.go:332","stacktrace":"go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/persistence/elasticsearch.(*visibilityMetricsClient).updateErrorMetric\n\t/temporal/common/persistence/elasticsearch/esVisibilityMetricClients.go:332\ngo.temporal.io/server/common/persistence/elasticsearch.(*visibilityMetricsClient).ListClosedWorkflowExecutions\n\t/temporal/common/persistence/elasticsearch/esVisibilityMetricClients.go:163\ngo.temporal.io/server/common/persistence.(*visibilityManagerWrapper).ListClosedWorkflowExecutions\n\t/temporal/common/persistence/visibilityWrapper.go:159\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).ListClosedWorkflowExecutions\n\t/temporal/service/frontend/workflowHandler.go:2550\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).ListClosedWorkflowExecutions.func2\n\t/temporal/service/frontend/dcRedirectionHandler.go:337\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/temporal/service/frontend/dcRedirectionPolicy.go:116\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).ListClosedWorkflowExecutions\n\t/temporal/service/frontend/dcRedirectionHandler.go:333\ngo.temporal.io/api/workflowservice/v1._WorkflowService_ListClosedWorkflowExecutions_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.4.0/workflowservice/v1/service.pb.go:1407\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/temporal/common/authorization/interceptor.go:136\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1051\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:100\ngoogle.golang.org/grpc.chainUnaryServerInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1037\ngo.temporal.io/api/workflowservice/v1._WorkflowService_ListClosedWorkflowExecutions_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.4.0/workflowservice/v1/service.pb.go:1409\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871"}

I am seeing this error in UI and couldn’t see much information.Can you suggest me on this error ?

2

How did you setup your Elasticsearch? Did you create index template and index prior running Temporal Server?

3

We have used this command to deploy temporal server.

helm install -f values/values.mysql.yaml temporaltest . --namespace live-staging --timeout 900s

and we are using our own mysql server.

It used to work before where we saw the workflows in UI, but all of the sudden we see this issue.
There is no change on the temporal deployment part from where we saw this issue.

4

With this command you will have your own Elasticsearch deployed within a Temporal cluster. Which is not good nor for prod neither for staging. Apparently your cluster restarted and you’ve lost your Elasticsearch index. Do you really need enhanced visibility (this is what Elasticsearch for)? If yes, I would recommend you to install Elasticsearch outside of your Temporal cluster (same way as you do for MySQL), configure it in values/values.elasticsearch.yaml file and pass it to the helm command. If not, just simply pass --set elasticsearch.enabled=false to the helm command and MySQL will be used for basic visibility.

5

Now we have redeployed the temporal server with --set elasticsearch.enabled=false
and we are seeing different error.
13 INTERNAL: ListClosedWorkflowExecutions failed. Error: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]

image
image3356×880 131 KB

Please let us know if we are doing something wrong here.

6

Standart visibility uses another MySQL database: temporal_visibility. It seems that you have configured visibility to use the same temporal database. But anyway you need to create visibility tables in the configured database.