Internal service error

Community Support
1

run workflow error， the workflow status is Running

{"level":"error","ts":"2021-02-24T19:45:06.812+0800","msg":"Internal service error","service":"frontend","wf-namespace":"default","wf-id":"4c4c5a2f-45e0-45bc-a05c-cb18db7944b0","wf-run-id":"87b4ebcc-15a1-4f7d-8100-b9f513e7819f","error":"corrupted data, nodeID cannot decrease","logging-call-at":"workflowHandler.go:3429","stacktrace":"go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error
	/root/go/src/github.com/temporalio/temporal/common/log/loggerimpl/logger.go:138
go.temporal.io/server/service/frontend.(*WorkflowHandler).error
	/root/go/src/github.com/temporalio/temporal/service/frontend/workflowHandler.go:3429
go.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue
	/root/go/src/github.com/temporalio/temporal/service/frontend/workflowHandler.go:875
go.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue.func2
	/root/go/src/github.com/temporalio/temporal/service/frontend/dcRedirectionHandler.go:532
go.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect
	/root/go/src/github.com/temporalio/temporal/service/frontend/dcRedirectionPolicy.go:116
go.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue
	/root/go/src/github.com/temporalio/temporal/service/frontend/dcRedirectionHandler.go:528
go.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1
	/root/go/pkg/mod/go.temporal.io/api@v1.2.1-0.20201207192846-11d32af78ad0/workflowservice/v1/service.pb.go:1083
go.temporal.io/server/common/authorization.(*interceptor).Interceptor
	/root/go/src/github.com/temporalio/temporal/common/authorization/interceptor.go:132
google.golang.org/grpc.getChainUnaryHandler.func1
	/root/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1051
go.temporal.io/server/common/rpc.ServiceErrorInterceptor
	/root/go/src/github.com/temporalio/temporal/common/rpc/grpc.go:100
google.golang.org/grpc.chainUnaryServerInterceptors.func1
	/root/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1037
go.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler
	/root/go/pkg/mod/go.temporal.io/api@v1.2.1-0.20201207192846-11d32af78ad0/workflowservice/v1/service.pb.go:1085
google.golang.org/grpc.(*Server).processUnaryRPC
	/root/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210
google.golang.org/grpc.(*Server).handleStream
	/root/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/root/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871"}
2

What version of the Temporal service and what DB are you using? If it is below 1.6 and the DB is Cassandra there was a corruption bug recently patched/fixed.

3

this one seems different.
Could you provide the following info:

  1. which DB & version: Cassandra 3.11 / MySQL 5.7 / PostgreSQL 9.6
  2. If Cassandra, what is the replication factor?
  3. Server version
4

server version: 1.5.1
db: mysql

5

db user mysql

6

could you come to our slack channel? it will be easier to debug your case

7

yes, thanks

8

synced offline, seems that the DB being used is not vanilla MySQL v5.7, but one which claims to be compatible.

For debugging purpose:

  1. please make sure DB is initialized correctly (not imported from an backup)
  2. please also run the load against your own DB setup vs a vanilla MySQL v5.7 for comparison