# Temporal in idle state generating huge read/write traffic to Cassandra **URL:** \ **Category:** Community Support\ **Tags:** java-sdk\ **Created:** [July 12, 2021, 10:58pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546 "2021-07-12T22:58:27Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![ramyamagham](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ramyamagham/32/1125_2.png) [@ramyamagham](https://community.temporal.io/u/ramyamagham)\ **Post date:** [July 12, 2021, 10:58pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/1 "2021-07-12T22:58:27Z")
Temporal in idle state is generating a huge read/write traffic to Cassandra. Not sure under what circumstances this starts happening but after running for a couple of days at 20K to 24K/Sec with Local Quorum. There were about 3% errors during this. Even when the incoming requests is stopped, the traffic keeps going on at 10K reads/sec almost an hour after the traffic is stopped. And when I start the traffic again, the total traffic on the DB goes back to 34K/sec. Only startToClose timeout is set to 1 second. Retry options are set to max attempts is at 3. ---
**Author:** ![Wenquan\_Xing](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/wenquan_xing/32/362_2.png) [@Wenquan\_Xing](https://community.temporal.io/u/Wenquan_Xing)\ **Post date:** [July 12, 2021, 11:02pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/2 "2021-07-12T23:02:39Z")
there are 2 cases: - timeout tasks, these tasks will be loaded from DB and processed even if workflow / activity has already gone - background task maintenance, e.g. periodically scan for new tasks in case write to DB timeouts. ---
**Author:** ![ramyamagham](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ramyamagham/32/1125_2.png) [@ramyamagham](https://community.temporal.io/u/ramyamagham)\ **Post date:** [July 12, 2021, 11:06pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/3 "2021-07-12T23:06:06Z")
How can I reduce the timeout tasks? The application does not need the tasks to be retried if it did not successfully finish in the first 5 seconds. So, I had set the scheduleToClose and scheduleToStart to maybe 5 seconds, will this case of 10K/sec be mitigated? Also we want to make sure the DB can handle any such background traffic, will it always be 10K/sec and/or how to throttle it? ---
**Author:** ![Wenquan\_Xing](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/wenquan_xing/32/362_2.png) [@Wenquan\_Xing](https://community.temporal.io/u/Wenquan_Xing)\ **Post date:** [July 12, 2021, 11:23pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/4 "2021-07-12T23:23:54Z")
> [@ramyamagham](#): > > How can I reduce the timeout tasks? by not setting a timeout, workflow can have no timeout if timeout set to 0 activity must have at least one timeout, e.g. start to close ---
**Author:** ![Wenquan\_Xing](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/wenquan_xing/32/362_2.png) [@Wenquan\_Xing](https://community.temporal.io/u/Wenquan_Xing)\ **Post date:** [July 12, 2021, 11:25pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/5 "2021-07-12T23:25:04Z")
> [@ramyamagham](#): > > Also we want to make sure the DB can handle any such background traffic, will it always be 10K/sec and/or how to throttle it? depends on your specific workflow (logic), timer (sleep) or timeout will always be persisted to DB and loaded for processing ---
**Author:** ![ramyamagham](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ramyamagham/32/1125_2.png) [@ramyamagham](https://community.temporal.io/u/ramyamagham)\ **Post date:** [July 12, 2021, 11:37pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/6 "2021-07-12T23:37:09Z")
Yes, I have startToClose timeout set to 1 second. My logic is to just try to perform the local\* activity in 1 second else discard it, as the calling client system through an API takes the error and goes away. There is no purpose for my application to retry. I am not sure about the timer you are referring to here, but I have retry interval as 1 second and total retry attempts as max 3. How can I avoid this huge local quorum and serial reads and writes as it is choking the Cassandra cluster. ---
**Author:** ![ramyamagham](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ramyamagham/32/1125_2.png) [@ramyamagham](https://community.temporal.io/u/ramyamagham)\ **Post date:** [July 12, 2021, 11:40pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/7 "2021-07-12T23:40:21Z")
![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/4/49f4a96ceead6900249f929fc279b0b03337dbd6.png) On the right panel I have attached the screenshot at table level, it shows 20K because it is not at co-ordinator level so it doubles, but it is 10K/sec local quorum reads happening primary on executions and history\_tree tables. ---
**Author:** ![ramyamagham](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ramyamagham/32/1125_2.png) [@ramyamagham](https://community.temporal.io/u/ramyamagham)\ **Post date:** [July 12, 2021, 11:42pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/8 "2021-07-12T23:42:26Z")
![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/f/f296181f1469703edfc8589607574650b5ba1595.png) Here is the write throughput which is also at table level and not co-ordinator level. The activity is mostly on executions and history node and tree tables. ---
**Author:** ![Wenquan\_Xing](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/wenquan_xing/32/362_2.png) [@Wenquan\_Xing](https://community.temporal.io/u/Wenquan_Xing)\ **Post date:** [July 13, 2021, 12:45am UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/9 "2021-07-13T00:45:27Z")
lots of read from history\_tree table does not seems to be right unless you have already run a load tests and finished workflow are being deleted. lots of write to history\_tree table seems to suggest you are still trying to start lots of new workflow. ---
**Author:** ![Wenquan\_Xing](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/wenquan_xing/32/362_2.png) [@Wenquan\_Xing](https://community.temporal.io/u/Wenquan_Xing)\ **Post date:** [July 13, 2021, 12:46am UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/10 "2021-07-13T00:46:32Z")
> [@ramyamagham](#): > > How can I avoid this huge local quorum and serial reads and writes as it is choking the Cassandra cluster. any mutation of workflow (create & update) will be translate to a LTW to cassandra delivering new workflow events will cause lots of read ---
**Author:** ![Wenquan\_Xing](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/wenquan_xing/32/362_2.png) [@Wenquan\_Xing](https://community.temporal.io/u/Wenquan_Xing)\ **Post date:** [July 13, 2021, 12:46am UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/11 "2021-07-13T00:46:51Z")
can you also provide these 2 metrics? - `poll_success` - `poll_success_sync` ---
**Author:** ![ramyamagham](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ramyamagham/32/1125_2.png) [@ramyamagham](https://community.temporal.io/u/ramyamagham)\ **Post date:** [July 13, 2021, 1:26am UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/12 "2021-07-13T01:26:25Z")
![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/6/6951ac222ecb5a279fbd05897ff52c6a35523bc7.png) poll\_success seems to be 0 when the traffic was stopped. The time between 23:00 to 00:00 you see and the other times there is no activity is when the traffic was paused. ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/8/8d24170da103af374f09070d90e3924c43adc17e.png) Here’s poll\_success\_sync which also looks similar to the above one. ---
**Author:** ![ramyamagham](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ramyamagham/32/1125_2.png) [@ramyamagham](https://community.temporal.io/u/ramyamagham)\ **Post date:** [July 13, 2021, 1:30am UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/13 "2021-07-13T01:30:48Z")
One more thing I’d like to add is that we have observed this behavior before and even if I left the system idle for the next 12 hours, the constant 10K/sec will continue to happen and they don’t come down. If need be, I can leave the system idle until tomorrow to prove it. ---
**Author:** ![Wenquan\_Xing](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/wenquan_xing/32/362_2.png) [@Wenquan\_Xing](https://community.temporal.io/u/Wenquan_Xing)\ **Post date:** [July 13, 2021, 7:48am UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/14 "2021-07-13T07:48:31Z")
> [@Wenquan\_Xing](#): > > - `poll_success` > - `poll_success_sync` these 2 metrics indicate how efficient is your setup, i.e. enough number of pollers or not, according to the metrics provided, this `poll_success_sync/poll_success` ratio is high enough. the 2 metrics below will show the async task processing `task / second`, e.g. processing of activity timers; user timers; workflow close, etc can you also provide these? ```auto sum by (operation) (rate(task_requests{operation=~"TransferActive.*"}[1m])) sum by (operation) (rate(task_requests{operation=~"TimerActive.*"}[1m])) ``` ---
**Author:** ![ramyamagham](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ramyamagham/32/1125_2.png) [@ramyamagham](https://community.temporal.io/u/ramyamagham)\ **Post date:** [July 13, 2021, 6:11pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/15 "2021-07-13T18:11:52Z")
Here are the two graphs for the past 2 days. The 1.6K requests/sec is the actual incoming traffic and we have stopped incoming traffic yesterday after my comment, as we wanted to see if the traffic would go down which is about 17 hours since when the traffic is stopped, and yet there are lingering reads/writes happening at the same rate. A couple of times the DB traffic went down for 30 minutes but it’s back at it again. ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/a/abb1a8ed835f639b5885a397458d8a36e977a265.png) ---
**Author:** ![ramyamagham](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ramyamagham/32/1125_2.png) [@ramyamagham](https://community.temporal.io/u/ramyamagham)\ **Post date:** [July 13, 2021, 6:12pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/16 "2021-07-13T18:12:06Z")
![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/0/070eba23460e2aa6863d1ac21d197a00018f9b6f.png) ---
**Author:** ![ramyamagham](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ramyamagham/32/1125_2.png) [@ramyamagham](https://community.temporal.io/u/ramyamagham)\ **Post date:** [July 13, 2021, 6:12pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/17 "2021-07-13T18:12:39Z")
DB read/write patterns in the last 20 hours: ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/b/b2853dedd966356152a140e029bb0b7481a126ef.png) ---
**Author:** ![ramyamagham](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ramyamagham/32/1125_2.png) [@ramyamagham](https://community.temporal.io/u/ramyamagham)\ **Post date:** [July 14, 2021, 5:54pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/18 "2021-07-14T17:54:55Z")
![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/3/3f0dfcb3195427ccda3c97b4d04abd4dad222cc9.png) After almost 27 hours the activity stopped, during all this time there was no incoming traffic. ---
**Author:** ![ramyamagham](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ramyamagham/32/1125_2.png) [@ramyamagham](https://community.temporal.io/u/ramyamagham)\ **Post date:** [July 14, 2021, 5:56pm UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/19 "2021-07-14T17:56:28Z")
![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/6/69a791eac2978311a357a6469dbbbbfe892b2249.png) Latest time active graph. cc: @Wenquan_Xing ---
**Author:** ![Wenquan\_Xing](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/wenquan_xing/32/362_2.png) [@Wenquan\_Xing](https://community.temporal.io/u/Wenquan_Xing)\ **Post date:** [July 20, 2021, 12:43am UTC](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546/20 "2021-07-20T00:43:09Z")
> [@ramyamagham](#): > > ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/0/070eba23460e2aa6863d1ac21d197a00018f9b6f.png) > > image1904×777 112 KB sorry about the delay, was sick in the past few days this graph shows that even after a workflow is finished, there will be other timeout related task still in DB pending processing, e.g. workflow timeout task, workflow cleanup task (delete the finished workflow after retention), etc. [Next page](https://community.temporal.io/t/temporal-in-idle-state-generating-huge-read-write-traffic-to-cassandra/2546.md?page=2)