RESOURCE_EXHAUSTED: namespace rate limit exceeded when countWorkflowExecutions and listWorkflowExecutions

Community Support
,
1

I encountered an error “RESOURCE_EXHAUSTED: namespace rate limit exceeded” when calling countWorkflowExecutions and listWorkflowExecutions using JavaSDK, but I called it very infrequently
( foreach 20 times to call once list and once count) like:

for(int i = 0; i<20; i++){
   listWorkflowExecutions(query);
   countWorkflowExecutions(query);
}

What can I do to get rid of this error?

My visibility store is elasticsearch.
Temporal server version is 1.20.3.
I am using the default configuration and no dynamicConfig is configured.

The following are the relevant temporal metrics:

sum(rate(service_errors_resource_exhausted{}[1m])) by (resource_exhausted_cause)

image
image1903×746 46.9 KB

sum(rate(service_requests{service_name=“frontend”, namespace=“<ns_name>”}[2m])) by (operation)
image
image1903×865 108 KB

I found some error logs what we don’t care about right now of frontend service:

{"level":"error","ts":"2024-01-24T09:33:01.371Z","msg":"Deprecation warning: 299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff \"Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.\"","logging-call-at":"logger.go:48","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error
	/home/builder/temporal/common/log/zap_logger.go:150
go.temporal.io/server/common/persistence/visibility/store/elasticsearch/client.(*errorLogger).Printf
	/home/builder/temporal/common/persistence/visibility/store/elasticsearch/client/logger.go:48
github.com/olivere/elastic/v7.(*Client).errorf
	/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.32/client.go:840
github.com/olivere/elastic/v7.(*Client).PerformRequest
	/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.32/client.go:1479
github.com/olivere/elastic/v7.(*CountService).Do
	/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.32/count.go:364
go.temporal.io/server/common/persistence/visibility/store/elasticsearch/client.(*clientImpl).Count
	/home/builder/temporal/common/persistence/visibility/store/elasticsearch/client/client_v7.go:218
go.temporal.io/server/common/persistence/visibility/store/elasticsearch.(*visibilityStore).CountWorkflowExecutions
	/home/builder/temporal/common/persistence/visibility/store/elasticsearch/visibility_store.go:571
go.temporal.io/server/common/persistence/visibility.(*visibilityManagerImpl).CountWorkflowExecutions
	/home/builder/temporal/common/persistence/visibility/visibility_manager_impl.go:255
go.temporal.io/server/common/persistence/visibility.(*visibilityManagerRateLimited).CountWorkflowExecutions
	/home/builder/temporal/common/persistence/visibility/visibility_manager_rate_limited.go:224
go.temporal.io/server/common/persistence/visibility.(*visibilityManagerMetrics).CountWorkflowExecutions
	/home/
builder/temporal/common/persistence/visibility/visiblity_manager_metrics.go:220
go.temporal.io/server/service/frontend.(*WorkflowHandler).CountWorkflowExecutions
	/home/builder/temporal/service/frontend/workflow_handler.go:2551
go.temporal.io/api/workflowservice/v1._WorkflowService_CountWorkflowExecutions_Handler.func1
	/go/pkg/mod/go.temporal.io/api@v1.18.2-0.20230324225508-f2c7ab685b44/workflowservice/v1/service.pb.go:1930
go.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1
	/home/builder/temporal/common/rpc/interceptor/retry.go:63
go.temporal.io/server/common/backoff.ThrottleRetryContext
	/home/builder/temporal/common/backoff/retry.go:199
go.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/retry.go:67
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/caller_info.go:80
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*SDKVersionInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/sdk_version.go:69
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/rate_limit.go:86
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/namespace_rate_limit.go:91
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor
.(*NamespaceCountLimitInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/namespace_count_limit.go:111
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).StateValidationIntercept
	/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:194
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/authorization.(*interceptor).Interceptor
	/home/builder/temporal/common/authorization/interceptor.go:158
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/telemetry.go:157
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/service/frontend.(*RedirectionInterceptor).handleRedirectAPIInvocation.func2
	/home/builder/temporal/service/frontend/redirection_interceptor.go:227
go.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect
	/home/builder/temporal/service/frontend/dcRedirectionPolicy.go:125
go.temporal.io/server/service/frontend.(*RedirectionInterceptor).handleRedirectAPIInvocation
	/home/builder/temporal/service/frontend/redirection_interceptor.go:224
go.temporal.io/server/service/frontend.(*RedirectionInterceptor).Intercept
	/home/builder/temporal/service/frontend/redirection_interceptor.go:184
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1
	/home/builder/temporal/common/metrics/grpc.go:66
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
g
o.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
	/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/namespace_logger.go:84
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept
	/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:111
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc.ServiceErrorInterceptor
	/home/builder/temporal/common/rpc/grpc.go:137
google.golang.org/grpc.chainUnaryInterceptors.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1155
go.temporal.io/api/workflowservice/v1._WorkflowService_CountWorkflowExecutions_Handler
	/go/pkg/mod/go.temporal.io/api@v1.18.2-0.20230324225508-f2c7ab685b44/workflowservice/v1/service.pb.go:1932
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966"}
2

These APIs are not designed for high read rates. Why do you need to call them frequently?

3

We have two scenarios about statistics and one about queries.

The first statistical scenario is: we want to count the failure rate grouped by WorkflowType within a certain period of time.

The second statistical scenario is: we want to count the reasons for failure grouped by WorkflowType within a certain period of time.

The query scenario is: We provide users with the ability to search WorkflowExecutions according to search attributes and count the number of searched workflows, just like temporal web ui.

Statistical scenarios are all low-frequency. Our current statistical method is to separately query failed executions according to each WorkflowType.
When it is necessary to count the reasons for failure, getWorkflowExecutionHistory(filter close event) will be called for each WorkflowExecution. The above problem occurs during list and count. So what is the list and count Api rate limit?

Query scenarios are concurrent, and the concurrency is controlled by the user, which may not exceed 100.

4

So the statistical scenarios look like a good fit for the existing feature.

The query scenario is not something I recommend. The list queries are not intended to scale the same way workflow executions scale. The solution is to use an external database with a schema tailored to your specific use case.

5

Thank you for your reply.
I would like to know what is the approximate call frequency limit of these APIs?
Because the namespace rate limit exceeded error occurs in the statistics scenario now, the statistics code is as follows:

List<String> workflowTypes = getStatWorkflowTypes()// 20 workflowTypes;
for(String workflowType: workflowTypes) {
      String query = queryWorkflowTypePeriodOfTimeExecutions;
      listApi(query);
      countApi(query);
}
6

I think it is about 10 per second. So, I would put a sleep between loop iterations.

7

Thank you for quick reply . I know how to do it.