Unable to submit workflow with "context deadline exceeded" error

Community Support
,
1

Hello!

I’ve encountered the context deadline exceeded error regularly while using the ExecuteWorkflow method.
Upon checking the frontend service logs, I can see the context deadline exceeded error with Unable to call matching.PollWorkflowTaskQueue. message around the same time as ExecuteWorkflow call.

temporalio/server:1.22.1

Call Workflow

workflowOptions := client.StartWorkflowOptions{
		ID:        "bq_" + jobUUID,
		TaskQueue: taskQueue,
		WorkflowExecutionErrorWhenAlreadyStarted: true,
	}
options := BQWorkflowOptions{
		JobID:     id,
	}
	we, err := w.temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, BQWorkflow, options)

FrontEnd Service logs:

logging_call_at	workflow_handler.go:897
msg: Unable to call matching.PollWorkflowTaskQueue.


stacktrace	
go.temporal.io/server/common/log.(*zapLogger).Error
	/home/builder/temporal/common/log/zap_logger.go:156
go.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue
	/home/builder/temporal/service/frontend/workflow_handler.go:897
go.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1
	/go/pkg/mod/go.temporal.io/api@v1.24.0/workflowservice/v1/service.pb.go:1679
go.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1
	/home/builder/temporal/common/rpc/interceptor/retry.go:63
go.temporal.io/server/common/backoff.ThrottleRetryContext
	/home/builder/temporal/common/backoff/retry.go:145
go.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/retry.go:67
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.57.0/server.go:1179
go.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/caller_info.go:80
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.57.0/server.go:1179
go.temporal.io/server/common/rpc/interceptor.(*SDKVersionInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/sdk_version.go:69
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.57.0/server.go:1179
go.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/rate_limit.go:88
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.57.0/server.go:1179
go.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/namespace_rate_limit.go:93
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.57.0/server.go:1179
go.temporal.io/server/common/rpc/interceptor.(*ConcurrentRequestLimitInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/concurrent_request_limit.go:121
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.57.0/server.go:1179
go.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).StateValidationIntercept
	/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:194
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.57.0/server.go:1179
go.temporal.io/server/common/authorization.(*interceptor).Interceptor
	/home/builder/temporal/common/authorization/interceptor.go:158

Could this be happening due to namespace rate limiting? (But reading through the SDK code, feel it might not be it).