# Difference between user and no user service latency metrics? **URL:** \ **Category:** Community Support\ **Tags:** history, metrics\ **Created:** [April 18, 2022, 10:36am UTC](https://community.temporal.io/t/difference-between-user-and-no-user-service-latency-metrics/4459 "2022-04-18T10:36:10Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![nitesh237](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/nitesh237/32/1656_2.png) [@nitesh237](https://community.temporal.io/u/nitesh237)\ **Post date:** [April 18, 2022, 10:36am UTC](https://community.temporal.io/t/difference-between-user-and-no-user-service-latency-metrics/4459/1 "2022-04-18T10:36:10Z")
Hi team, Can someone help with the difference between these two metrics? ```auto service_latency_nouserlatency service_latency_userlatency ``` Context: We are trying to conduct load testing and seems like `service_latency_nouserlatency` is causing the major bottleneck in the history service. Understanding the metrics will help us identifying the bottleneck in the cluster. ~Nitesh ---
**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\ **Post date:** [April 18, 2022, 11:05pm UTC](https://community.temporal.io/t/difference-between-user-and-no-user-service-latency-metrics/4459/3 "2022-04-18T23:05:26Z")
> service\_latency\_userlatency This metric measures latency to obtain the per-workflow lock. > service\_latency\_nouserlatency This latency is the overall latency minus service\_latency\_userlatency > seems like `service_latency_nouserlatency` is causing the major bottleneck in the history service This can be caused by workflows scheduling too many activities concurrently. Are there any other metrics that stick out as there could be other causes as well. Any error logs on db or history service? ---
**Author:** ![Yimin\_Chen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/yimin_chen/32/963_2.png) [@Yimin\_Chen](https://community.temporal.io/u/Yimin_Chen)\ **Post date:** [April 19, 2022, 4:26am UTC](https://community.temporal.io/t/difference-between-user-and-no-user-service-latency-metrics/4459/4 "2022-04-19T04:26:25Z")
Which API calls is experiencing the high latency? In general, first to check is persistence\_latency, specifically look for operation=“UpdateWorkflowExecution”. If you see persistence\_latency is high, it means you need to scale out your persistence. Second is to check your CPU usage on your history service instance, are they saturate CPU limit. ---
**Author:** ![nitesh237](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/nitesh237/32/1656_2.png) [@nitesh237](https://community.temporal.io/u/nitesh237)\ **Post date:** [April 19, 2022, 4:42am UTC](https://community.temporal.io/t/difference-between-user-and-no-user-service-latency-metrics/4459/5 "2022-04-19T04:42:29Z")
Thanks, folks for the replies. These were useful. So, I suspected the history service CPU to be the bottleneck and ran the same benchmarking with 2x nodes. The CPU utilization for history service pods seems to be in range 40-50% range now. However, I still see a difference in service latency vs DB latency as can be seen below. @tihomir to answer your question task queue latency(both `task_latency_queue` and `task_latency_queue_nouserlatency`) and task processing latency(`task_latency_processing`) seem to be sticking out. Do we know what could lead to this ? The persistence latency seems to be in decent range here @Yimin_Chen. ![Screenshot 2022-04-19 at 11.57.47 AM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/c/cd9200014ba81247c21a9c6d5e8be5e5e8fa19ab.jpeg) Based on this I checked my workflow/activity workers CPU utilization which was in the 10-15% range. Is there anything else I should be looking into ? ---
**Author:** ![Yimin\_Chen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/yimin_chen/32/963_2.png) [@Yimin\_Chen](https://community.temporal.io/u/Yimin_Chen)\ **Post date:** [April 19, 2022, 3:55pm UTC](https://community.temporal.io/t/difference-between-user-and-no-user-service-latency-metrics/4459/6 "2022-04-19T15:55:23Z")
Could you group the service\_latency metric by operation? Also check if there is any service\_errors metrics? (right now there are different service\_errors\_xxx for different type of errors, but we will consolidate them into one metric with different error type tag.) ---
**Author:** ![nitesh237](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/nitesh237/32/1656_2.png) [@nitesh237](https://community.temporal.io/u/nitesh237)\ **Post date:** [April 19, 2022, 7:56pm UTC](https://community.temporal.io/t/difference-between-user-and-no-user-service-latency-metrics/4459/7 "2022-04-19T19:56:12Z")
Hi Yimin, My bad PFA the metrics grouped by operations. ![Screenshot 2022-04-20 at 12.55.05 AM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/4/4545369ed0ea03970ed32070afed344da493bd4d.jpeg) Found error graph for the following: - `service_errors_entity_not_found` ![Screenshot 2022-04-20 at 1.21.49 AM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/e/ef05d26f12cb57e5f925ef1435d2f219201dbea1.png) - `client_errors` - `service_errors_resource_exhausted` from frontend ---
**Author:** ![Yimin\_Chen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/yimin_chen/32/963_2.png) [@Yimin\_Chen](https://community.temporal.io/u/Yimin_Chen)\ **Post date:** [April 20, 2022, 5:08pm UTC](https://community.temporal.io/t/difference-between-user-and-no-user-service-latency-metrics/4459/8 "2022-04-20T17:08:40Z")
service\_errors\_resource\_exhausted is what is causing trouble now. Look for the resource\_exhausted\_cause of that metric. There are 2 types of reasons: RpsLimit or ConcurrentLimit. Both are dynamic configured at frontend. frontend.rps → rps limit overall frontend.namespaceRPS → rps limit per namespace frontend.namespaceCount → concurrent poller limit per namespace Adjust these 3 configs. They are all per frontend instance limit. ---
**Author:** ![nitesh237](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/nitesh237/32/1656_2.png) [@nitesh237](https://community.temporal.io/u/nitesh237)\ **Post date:** [April 28, 2022, 5:10am UTC](https://community.temporal.io/t/difference-between-user-and-no-user-service-latency-metrics/4459/10 "2022-04-28T05:10:50Z")
Hey folks thanks for your response. > [@tihomir](#): > > This can be caused by workflows scheduling too many activities concurrently. Are there any other metrics that stick out as there could be other causes as well. Any error logs on db or history service? Turns out this was the root cause of the problem. The workflow specs with which I was benchmarking were producing too many activities concurrently. I configured it to the write specs and it was working as expected. To reduce the resource exhausted errors in FE layer, I limited the number of task pollers per workers and it worked fine post that.