Possible cause of latency difference between History Server Backend and History Server gRPC Client in Frontend Server?

[pratyay]

We are seeing significant difference between server side and perceived latencies at client end for RespondWorkflowTaskCompleted task.
Will this need connection pool tuning? What other reasons could be there for this difference?

History Server Dashboard Reporting p99 ~ 120ms

History Client in Frontend Reporting p99 ~ 5s

[taonic]

We have seen CPU throttling on the frontend causing the mismatched latency like this.