What is Pollactivity/workflow taskqueue RPS limit error on Frontend?

Community Support
, ,
1

Hi team,

We are getting pollactivity/workflowtaskqueue RPS error in Frontend service, while we have set Frontend rps as 12000 in dynamic config, while we are getting this error at 6000 frontend request levels.
Also the error(resourse_exhausted - rps limit) is only on pollactivitytaskqueue / pollworkflowtaskqueue operations in frontend service.
Dynamic config:

image (2)
image (2)2194×1434 184 KB

Request count and errors :
image
image2920×614 129 KB

Worker options config :
image
image1540×338 55.8 KB

note : we haven’t set any taskqueuepersecond parameter at workeroptions (ref: workflowtaskqueue)

We are unable to drilldown on this error further, could someone help us in narrowing down this issue.
thanks in advance.

2

whats the server version used? how many frontend pods to you stand up?

3

Temporal server version used - 2.31.2, 3 Frontend pods were up during this load.

4

Temporal server version used - 2.31.2

Apologies for the mistake in Temporal server version used.
Temporal server version - 1.25.2, Temporal UI version - 2.31.2.

5

@tihomir , Can you provide any information on this, why its happening?

6

Frontend pods were up during this load.

how many frontend pods? do you have a load-balancer infront of them, if so can you check if its evenly distributing load across all frontends?

frontend.namespaceRps is per-frontend config
you could use frontend.globalNamespaceRPS which would be evenly distribute across your frontends

why its happening?

Rps has limit and burst. you might wanna try try increase frontend.namespaceBurstRatio from default 2 to larger value if your nave pretty bursty use case and see if that helps too

2 Likes