# CPU Utilization with AWS RDS (MySQL) **URL:** \ **Category:** Community Support\ **Tags:** mysql\ **Created:** [October 27, 2021, 3:24pm UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252 "2021-10-27T15:24:35Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Lam\_Tran](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/lam_tran/32/1382_2.png) [@Lam\_Tran](https://community.temporal.io/u/Lam_Tran)\ **Post date:** [October 27, 2021, 3:24pm UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/1 "2021-10-27T15:24:35Z")
Hi, has anyone faced an issue with CPU utilization in AWS RDS ? Here is my setup - Temporal v1.13.0 - RDS instance db.t3.small, running MySQL 8 (I tried MySQL 5.7 in RDS but still getting high cpu) - No traffic comes in, just deploy Temporal on 3 EC2 nodes (via Nomad /containers) - I used mysqladmin status command and saw that the number of queries per seconds is not low I think Uptime: 24470 Threads: 16 Questions: 15216291 Slow queries: 0 Opens: 907 Flush tables: 1 Open tables: 532 Queries per second avg: 621.834 - The output of processlist is below ```auto mysql> show full processlist; +------+-------------+-------------------+----------+---------+------+----------+-----------------------+ | Id | User | Host | db | Command | Time | State | Info | +------+-------------+-------------------+----------+---------+------+----------+-----------------------+ | 4 | rdsadmin | localhost:29260 | NULL | Sleep | 2 | | NULL | | 2599 | mysql5_root | 10.64.7.18:46378 | NULL | Query | 0 | starting | show full processlist | | 2935 | temporal | 10.64.7.156:40408 | temporal | Sleep | 16 | | NULL | | 2940 | temporal | 10.64.7.211:37512 | temporal | Sleep | 3 | | NULL | | 2941 | temporal | 10.64.7.156:40492 | temporal | Sleep | 6 | | NULL | | 2942 | temporal | 10.64.7.211:37561 | temporal | Sleep | 6 | | NULL | | 2943 | temporal | 10.64.7.211:37560 | temporal | Sleep | 6 | | NULL | | 2944 | temporal | 10.64.7.211:37564 | temporal | Sleep | 3 | | NULL | | 2945 | temporal | 10.64.7.172:35308 | temporal | Sleep | 3 | | NULL | | 2946 | temporal | 10.64.7.172:35428 | temporal | Sleep | 7 | | NULL | | 2947 | temporal | 10.64.7.172:35426 | temporal | Sleep | 7 | | NULL | | 2948 | temporal | 10.64.7.172:35430 | temporal | Sleep | 7 | | NULL | | 2949 | temporal | 10.64.7.172:35432 | temporal | Sleep | 1 | | NULL | | 2950 | temporal | 10.64.7.172:35450 | temporal | Sleep | 8 | | NULL | | 2951 | temporal | 10.64.7.172:35458 | temporal | Sleep | 17 | | NULL | | 2952 | temporal | 10.64.7.172:35460 | temporal | Sleep | 8 | | NULL | | 2953 | temporal | 10.64.7.172:35464 | temporal | Sleep | 7 | | NULL | | 2954 | temporal | 10.64.7.172:35462 | temporal | Sleep | 7 | | NULL | | 2955 | temporal | 10.64.7.172:35466 | temporal | Sleep | 17 | | NULL | | 2956 | temporal | 10.64.7.156:40732 | temporal | Sleep | 3 | | NULL | | 2957 | temporal | 10.64.7.156:40734 | temporal | Sleep | 7 | | NULL | | 2958 | temporal | 10.64.7.156:40736 | temporal | Sleep | 3 | | NULL | | 2959 | temporal | 10.64.7.156:40738 | temporal | Sleep | 7 | | NULL | | 2960 | temporal | 10.64.7.156:40752 | temporal | Sleep | 14 | | NULL | | 2961 | temporal | 10.64.7.211:37794 | temporal | Sleep | 6 | | NULL | | 2962 | temporal | 10.64.7.156:40810 | temporal | Sleep | 7 | | NULL | | 2963 | temporal | 10.64.7.156:40812 | temporal | Sleep | 7 | | NULL | | 2964 | temporal | 10.64.7.156:40814 | temporal | Sleep | 6 | | NULL | | 2965 | temporal | 10.64.7.211:37870 | temporal | Sleep | 2 | | NULL | | 2966 | temporal | 10.64.7.211:37886 | temporal | Sleep | 2 | | NULL | | 2967 | temporal | 10.64.7.211:37890 | temporal | Sleep | 2 | | NULL | | 2968 | temporal | 10.64.7.211:37892 | temporal | Sleep | 2 | | NULL | | 2969 | temporal | 10.64.7.211:37894 | temporal | Sleep | 2 | | NULL | +------+-------------+-------------------+----------+---------+------+----------+-----------------------+ ``` - I also use another MySQL 8 instance running in GCP CloudSQL but don’t see this problem. Can you guys suggest the minimum resources (cpu/ram) to run the Temporal on MySQL ? Thanks ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/0/0716554bb6d8c3793702b3cd40e4dc414fa60936.png) ---
**Author:** ![longndb](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/longndb/32/1441_2.png) [@longndb](https://community.temporal.io/u/longndb)\ **Post date:** [October 28, 2021, 3:58am UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/2 "2021-10-28T03:58:09Z")
In the same issue ---
**Author:** ![Yimin\_Chen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/yimin_chen/32/963_2.png) [@Yimin\_Chen](https://community.temporal.io/u/Yimin_Chen)\ **Post date:** [October 28, 2021, 5:37am UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/3 "2021-10-28T05:37:52Z")
@Lam_Tran , do you have the top queries that were executed? Or could you share the persistence metrics group by operations? If you are using grafana/promtheus, the query for metrics would looks something like: `sum by (operation) (rate(persistence_requests[1m]))`. Or better yet, if you could take a goroutine dump and share with us, that will greatly help us quickly identify the problem. Thank you. ---
**Author:** ![Lam\_Tran](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/lam_tran/32/1382_2.png) [@Lam\_Tran](https://community.temporal.io/u/Lam_Tran)\ **Post date:** [October 28, 2021, 6:10am UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/4 "2021-10-28T06:10:22Z")
Hi @Yimin_Chen I am checking with my team to get some metrics from RDS (currently we are using Datadog to view) Btw, how to enable pprof to take the goroutine dump ? I am using Nomad to deploy the Temporal as containers, the way I provision configuration for Temporal is the ENV from this template: [temporal/config\_template.yaml at master · temporalio/temporal · GitHub](https://github.com/temporalio/temporal/blob/master/docker/config_template.yaml). This does not have pprof configuration. Thanks ---
**Author:** ![Lam\_Tran](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/lam_tran/32/1382_2.png) [@Lam\_Tran](https://community.temporal.io/u/Lam_Tran)\ **Post date:** [October 28, 2021, 6:18am UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/5 "2021-10-28T06:18:53Z")
Hi @Yimin_Chen Here is some metrics from our Datadog Agent ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/6/69d02ce2499cad17b324b4ee867362dc2b565be1.png) ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/8/8b3eed1a4c7db53d8007a740e3b41353816f2426.png) ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/d/d4fe3656a5943964dc08c1aaf3118a87a88e3176.png) I think that the select rate is too high which can lead to the cpu utilization problem. ---
**Author:** ![Lam\_Tran](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/lam_tran/32/1382_2.png) [@Lam\_Tran](https://community.temporal.io/u/Lam_Tran)\ **Post date:** [October 28, 2021, 6:26am UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/6 "2021-10-28T06:26:38Z")
We also have another RDS instance (db.t3.large) and it runs MySQL 5.7. There are some metrics getting from Performance Insight of AWS RDS, hope this can help ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/6/6227da862d05ea28d6ff10c0e3b744a29f6f06bc.jpeg) ---
**Author:** ![alex](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/alex/32/15_2.png) [@alex](https://community.temporal.io/u/alex)\ **Post date:** [October 28, 2021, 7:25pm UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/7 "2021-10-28T19:25:13Z")
Unfortunately `pprof` is disabled by default in production images and even removed from config completely. To make it work you need to modify generated `config/docker.yaml` manually or modify `config_template.yaml` and regenerate config with `dockerize`. You need to modify `global` section by adding these [two lines](https://github.com/temporalio/temporal/blob/master/config/development.yaml#L18-L19): ```auto global: pprof: port: 7936 ``` ---
**Author:** ![Yimin\_Chen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/yimin_chen/32/963_2.png) [@Yimin\_Chen](https://community.temporal.io/u/Yimin_Chen)\ **Post date:** [October 28, 2021, 7:43pm UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/8 "2021-10-28T19:43:27Z")
Once you have the pprof port setup, you can take a goroutine dump by open a port forwarding to your container: ```bash kubectl port-forward your_temporal_history_pod_id 7936:7936 -n your_kubenate_namespace ``` Then, you can fetch a goroutine dump by: ```bash wget 'http://localhost:7936/debug/pprof/goroutine?debug=2' -O goroutine.txt ``` ---
**Author:** ![Lam\_Tran](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/lam_tran/32/1382_2.png) [@Lam\_Tran](https://community.temporal.io/u/Lam_Tran)\ **Post date:** [October 29, 2021, 6:34am UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/9 "2021-10-29T06:34:18Z")
Hi @alex and @Yimin_Chen Here is the goroutine.txt: > ---
**Author:** ![alex](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/alex/32/15_2.png) [@alex](https://community.temporal.io/u/alex)\ **Post date:** [October 29, 2021, 6:39am UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/10 "2021-10-29T06:39:47Z")
From the first sight nothing bad there. Let me double check. After upgrade to 1.13, you see high (almost 100%) CPU load on your MySQL nodes and Temporal executes something like 6k SELECT queries per second. Did I get it right? ---
**Author:** ![Lam\_Tran](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/lam_tran/32/1382_2.png) [@Lam\_Tran](https://community.temporal.io/u/Lam_Tran)\ **Post date:** [October 29, 2021, 6:43am UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/11 "2021-10-29T06:43:36Z")
Thanks @alex Actually I see the cpu load issue since v1.12.3 and after upgrading to v1.13.0, it’s still there. ---
**Author:** ![alex](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/alex/32/15_2.png) [@alex](https://community.temporal.io/u/alex)\ **Post date:** [October 29, 2021, 6:44am UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/12 "2021-10-29T06:44:59Z")
ok. This definitely will help. What was the last version you tried and it didn’t have it? ---
**Author:** ![Lam\_Tran](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/lam_tran/32/1382_2.png) [@Lam\_Tran](https://community.temporal.io/u/Lam_Tran)\ **Post date:** [October 29, 2021, 6:49am UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/13 "2021-10-29T06:49:03Z")
From the 1st time I used Temporal (1.11.x) on AWS RDS, the CPU load issue already happened, it actually consumes the CPU of RDS much, by time, not peak immediately (even using MariaDB 10.3, MySQL 5.7 and MySQL 8.0 on different instance types) When I compare to the same DB instance type using MySQL 8 on GCP CloudSQL, there is no issue (Temporal 1.12 and 1.13), CPU ~ 10-20% (instance has 1 vCPU) ---
**Author:** ![Yimin\_Chen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/yimin_chen/32/963_2.png) [@Yimin\_Chen](https://community.temporal.io/u/Yimin_Chen)\ **Post date:** [October 29, 2021, 7:08pm UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/14 "2021-10-29T19:08:22Z")
How many pods for frontend/history/matching/worker do you have in your setup? How many history shard do you have in your config? (From the goroutine count, it seems like default 4 shard). And from the goroutine dump, it seems you run all 4 service role frontend/history/matching/worker in one process? ---
**Author:** ![Lam\_Tran](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/lam_tran/32/1382_2.png) [@Lam\_Tran](https://community.temporal.io/u/Lam_Tran)\ **Post date:** [October 30, 2021, 2:37am UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/15 "2021-10-30T02:37:44Z")
Hi @Yimin_Chen My setup is: - 4 services frontend/history/matching/worker are running in one process, I use image temporalio/server:1.13.0 and do not update the SERVICES env - There are 3 nodes (EC2 on AWS), each node will run one process as a container (deployed via Nomad) - Default shard is used (4) ---
**Author:** ![Yimin\_Chen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/yimin_chen/32/963_2.png) [@Yimin\_Chen](https://community.temporal.io/u/Yimin_Chen)\ **Post date:** [October 30, 2021, 4:08pm UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/16 "2021-10-30T16:08:58Z")
@Lam_Tran , from the goroutine dump, there is nothing out of ordinary. In fact, there is only one goroutine with call stack in persistence layer and it is idle on purge timer for namespace replication task. There is literally no work happening on the persistence layer when this dump was taken. Is it possible that our of your 3 temporal node, only one of them is going crazy, but was not the one you captured the dump? Also, is it possible to get some sample queries that was executed by your database? And did you setup metrics, it would be very useful to see which persistence API was called by temproal. Sample query for promethus would be `sum by (operation) (rate(persistence_requests[1m]))` . ---
**Author:** ![Lam\_Tran](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/lam_tran/32/1382_2.png) [@Lam\_Tran](https://community.temporal.io/u/Lam_Tran)\ **Post date:** [November 1, 2021, 4:21am UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/17 "2021-11-01T04:21:26Z")
Hi @Yimin_Chen Let me share 3 goroutines dump output from 3 nodes first. For the database metrics and application metrics from Prometheus, I will share soon, after finishing the setup. Node 0: [node-0 · GitHub](https://gist.github.com/ngoclamtran/1301f7fd31aa7e30791d75a5b75cd473) Node 1: [node-1 · GitHub](https://gist.github.com/ngoclamtran/b7b29a96596954979d9f2acd40de0b39) Node 2: [node-2 · GitHub](https://gist.github.com/ngoclamtran/ab179b0811b9332dbe9b45ed9f405f77) ---
**Author:** ![Lam\_Tran](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/lam_tran/32/1382_2.png) [@Lam\_Tran](https://community.temporal.io/u/Lam_Tran)\ **Post date:** [November 1, 2021, 10:03am UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/18 "2021-11-01T10:03:18Z")
Hi @Yimin_Chen Here is the metrics grouping by the operation in Prometheus. From this we can see that the count of GetTask operation is very high. ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/8/85985854095280ad4ed20daf2f99466b573314d6.png) ---
**Author:** ![Lam\_Tran](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/lam_tran/32/1382_2.png) [@Lam\_Tran](https://community.temporal.io/u/Lam_Tran)\ **Post date:** [November 1, 2021, 4:04pm UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/19 "2021-11-01T16:04:00Z")
Hi @Yimin_Chen I have built another AWS RDS instance using Postgresql and deploy 3 instances of Temporal on it (same deployment stack via Nomad, each of container on a node uses image temporalio/server). The result from Prometheus looks different: ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/e/efaf3aaee46a956851559b131a2f85ab791e19de.jpeg) ---
**Author:** ![Yimin\_Chen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/yimin_chen/32/963_2.png) [@Yimin\_Chen](https://community.temporal.io/u/Yimin_Chen)\ **Post date:** [November 1, 2021, 4:13pm UTC](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252/20 "2021-11-01T16:13:17Z")
@Lam_Tran thank you for the info, we will dig into this shortly and get back to you once we find anything. [Next page](https://community.temporal.io/t/cpu-utilization-with-aws-rds-mysql/3252.md?page=2)