RESOURCE_EXHAUSTED: namespace rate limit exceeded for cron Job

Community Support
1

Hi Maxim/Thiomir

I am enjoying working with temporal, but now I am stucked with one issue i.e.

RESOURCE_EXHAUSTED: namespace rate limit exceeded

And I am getting above error for explained implementation

I am having a CRON job which will read a CSV file from SFTP containing n number of data, and by iterating each row from CSV file i will signal to its corresponding running workflow based on workflow id(random id: temporal generated)

For 5-10 records it does not gave me any error but when number of lines in CSV file went above 50 suddenly this error occured resource exhausted.

My implementation is like

  1. Inside workflow impl i am calling read activity to read records from that CSV file.
  2. Inside read activity i am calling builder query to get ListWorkflowExecutionInfo and mapping that data to Java map and returns Java map as a data.
  3. on that map i am iterating for each data and sending signal to workflow using external workflow stub for each record

map forEach( data → { send signal using external workflow stub})

How can I get rid of this error.

3

RESOURCE_EXHAUSTED: namespace rate limit exceeded

  1. Inside read activity i am calling builder query to get ListWorkflowExecutionInfo and mapping that data to Java map and returns Java map as a data.

These are related, visibility api have own rps limits. Dynamic config knob:

frontend.namespaceRPS.visibility (default 50)

Seems you are being rate-limited on it. Try increasing this value or consider adding more frontend containers/pods as this is a per-instance config.

Regarding your use case, can you give more info on this? I don’t think having to query visibility api here is necessary, but knowing more would help to understand. Thanks.

4

Just to add, if you increase frontend.namespaceRPS.visibility and still run into rate limits, try also increasing frontend.namespaceBurst.visibility