RESOURCE_EXHAUSTED: namespace rate limit exceeded when countWorkflowExecutions and listWorkflowExecutions

zuodimiaoyun · January 24, 2024, 11:49am

I encountered an error “RESOURCE_EXHAUSTED: namespace rate limit exceeded” when calling countWorkflowExecutions and listWorkflowExecutions using JavaSDK, but I called it very infrequently
( foreach 20 times to call once list and once count) like:

for(int i = 0; i<20; i++){
   listWorkflowExecutions(query);
   countWorkflowExecutions(query);
}

What can I do to get rid of this error?

My visibility store is elasticsearch.
Temporal server version is 1.20.3.
I am using the default configuration and no dynamicConfig is configured.

The following are the relevant temporal metrics:

sum(rate(service_errors_resource_exhausted{}[1m])) by (resource_exhausted_cause)

sum(rate(service_requests{service_name=“frontend”, namespace=“<ns_name>”}[2m])) by (operation)

I found some error logs what we don’t care about right now of frontend service:

{"level":"error","ts":"2024-01-24T09:33:01.371Z","msg":"Deprecation warning: 299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff \"Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.\"","logging-call-at":"logger.go:48","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error
	/home/builder/temporal/common/log/zap_logger.go:150
go.temporal.io/server/common/persistence/visibility/store/elasticsearch/client.(*errorLogger).Printf
	/home/builder/temporal/common/persistence/visibility/store/elasticsearch/client/logger.go:48
github.com/olivere/elastic/v7.(*Client).errorf
	/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.32/client.go:840
github.com/olivere/elastic/v7.(*Client).PerformRequest
	/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.32/client.go:1479
github.com/olivere/elastic/v7.(*CountService).Do
	/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.32/count.go:364
go.temporal.io/server/common/persistence/visibility/store/elasticsearch/client.(*clientImpl).Count
	/home/builder/temporal/common/persistence/visibility/store/elasticsearch/client/client_v7.go:218
go.temporal.io/server/common/persistence/visibility/store/elasticsearch.(*visibilityStore).CountWorkflowExecutions
	/home/builder/temporal/common/persistence/visibility/store/elasticsearch/visibility_store.go:571
go.temporal.io/server/common/persistence/visibility.(*visibilityManagerImpl).CountWorkflowExecutions
	/home/builder/temporal/common/persistence/visibility/visibility_manager_impl.go:255
go.temporal.io/server/common/persistence/visibility.(*visibilityManagerRateLimited).CountWorkflowExecutions
	/home/builder/temporal/common/persistence/visibility/visibility_manager_rate_limited.go:224
go.temporal.io/server/common/persistence/visibility.(*visibilityManagerMetrics).CountWorkflowExecutions
	/home/
builder/temporal/common/persistence/visibility/visiblity_manager_metrics.go:220
go.temporal.io/server/service/frontend.(*WorkflowHandler).CountWorkflowExecutions
	/home/builder/temporal/service/frontend/workflow_handler.go:2551
go.temporal.io/api/workflowservice/v1._WorkflowService_CountWorkflowExecutions_Handler.func1
	/go/pkg/mod/go.temporal.io/api@v1.18.2-0.20230324225508-f2c7ab685b44/workflowservice/v1/service.pb.go:1930
go.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1
	/home/builder/temporal/common/rpc/interceptor/retry.go:63
go.temporal.io/server/common/backoff.ThrottleRetryContext
	/home/builder/temporal/common/backoff/retry.go:199
go.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/retry.go:67
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/caller_info.go:80
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*SDKVersionInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/sdk_version.go:69
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/rate_limit.go:86
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/namespace_rate_limit.go:91
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor
.(*NamespaceCountLimitInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/namespace_count_limit.go:111
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).StateValidationIntercept
	/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:194
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/authorization.(*interceptor).Interceptor
	/home/builder/temporal/common/authorization/interceptor.go:158
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/telemetry.go:157
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/service/frontend.(*RedirectionInterceptor).handleRedirectAPIInvocation.func2
	/home/builder/temporal/service/frontend/redirection_interceptor.go:227
go.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect
	/home/builder/temporal/service/frontend/dcRedirectionPolicy.go:125
go.temporal.io/server/service/frontend.(*RedirectionInterceptor).handleRedirectAPIInvocation
	/home/builder/temporal/service/frontend/redirection_interceptor.go:224
go.temporal.io/server/service/frontend.(*RedirectionInterceptor).Intercept
	/home/builder/temporal/service/frontend/redirection_interceptor.go:184
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1
	/home/builder/temporal/common/metrics/grpc.go:66
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
g
o.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
	/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept
	/home/builder/temporal/common/rpc/interceptor/namespace_logger.go:84
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept
	/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:111
google.golang.org/grpc.getChainUnaryHandler.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164
go.temporal.io/server/common/rpc.ServiceErrorInterceptor
	/home/builder/temporal/common/rpc/grpc.go:137
google.golang.org/grpc.chainUnaryInterceptors.func1
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1155
go.temporal.io/api/workflowservice/v1._WorkflowService_CountWorkflowExecutions_Handler
	/go/pkg/mod/go.temporal.io/api@v1.18.2-0.20230324225508-f2c7ab685b44/workflowservice/v1/service.pb.go:1932
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966"}

maxim · January 25, 2024, 12:45am

These APIs are not designed for high read rates. Why do you need to call them frequently?

zuodimiaoyun · January 25, 2024, 2:57am

We have two scenarios about statistics and one about queries.

The first statistical scenario is: we want to count the failure rate grouped by WorkflowType within a certain period of time.

The second statistical scenario is: we want to count the reasons for failure grouped by WorkflowType within a certain period of time.

The query scenario is: We provide users with the ability to search WorkflowExecutions according to search attributes and count the number of searched workflows, just like temporal web ui.

Statistical scenarios are all low-frequency. Our current statistical method is to separately query failed executions according to each WorkflowType.
When it is necessary to count the reasons for failure, getWorkflowExecutionHistory(filter close event) will be called for each WorkflowExecution. The above problem occurs during list and count. So what is the list and count Api rate limit?

Query scenarios are concurrent, and the concurrency is controlled by the user, which may not exceed 100.

maxim · January 26, 2024, 12:51am

So the statistical scenarios look like a good fit for the existing feature.

The query scenario is not something I recommend. The list queries are not intended to scale the same way workflow executions scale. The solution is to use an external database with a schema tailored to your specific use case.

zuodimiaoyun · January 26, 2024, 2:08am

Thank you for your reply.
I would like to know what is the approximate call frequency limit of these APIs?
Because the namespace rate limit exceeded error occurs in the statistics scenario now, the statistics code is as follows:

List<String> workflowTypes = getStatWorkflowTypes()// 20 workflowTypes;
for(String workflowType: workflowTypes) {
      String query = queryWorkflowTypePeriodOfTimeExecutions;
      listApi(query);
      countApi(query);
}

maxim · January 26, 2024, 2:09am

I think it is about 10 per second. So, I would put a sleep between loop iterations.

zuodimiaoyun · January 26, 2024, 2:11am

Thank you for quick reply . I know how to do it.

Topic		Replies	Views
Strange UI bug for advanced query Community Support	7	467	December 30, 2020
Timeouts on visibility-tasks but workflows still completing Community Support	3	388	January 11, 2023
VisibilityUpsertExecution error unable to search attribute hogging temporal queue and causing other requests to temporal fail Community Support elasticsearch , retries , database	4	714	March 21, 2022
Elasticsearch - ListWorkflowExecutions failed - No mapping found for [CloseTime] Community Support elasticsearch , advanced_visibility	6	559	September 16, 2023
Error: "Search attributes size exceeds limits. Fail workflow." on Self hosted Temporalite server Community Support go-sdk , general-impl , metrics	2	273	July 24, 2023

RESOURCE_EXHAUSTED: namespace rate limit exceeded when countWorkflowExecutions and listWorkflowExecutions

Related Topics