RESOURCE_EXHAUSTED: namespace rate limit exceeded

Community Support
,
15

Could you check all service requests going through the frontend service for the namespace your running the workflows on: sum(rate(service_requests{service_name="frontend", namespace="<ns_name>"}[2m])) by (operation)

frontend:

image
image1872×843 225 KB

Also, are you also setting frontend.rps in dynamic config? (default 2400) This is the frontend overall rps limit.

Yes, below is from our dynamic config.

frontend.rps:
- value: 76800
  constraints: {}
frontend.namespaceRPS:
- value: 76800
  constraints: {}

service requests for history was roughly the same as frontend, and matching was roughly double at times (I can only include one image per post as I am still new to the forum).

We have increased rps for matching and history before the same load test.

matching.rps:
- value: 76800
  constraints: {}
history.rps:
- value: 76800
  constraints: {}