# History\_node keeps growing **URL:** \ **Category:** Community Support\ **Tags:** postgresql\ **Created:** [September 12, 2022, 1:37pm UTC](https://community.temporal.io/t/history-node-keeps-growing/5944 "2022-09-12T13:37:57Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Kani](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/kani/32/2504_2.png) [@Kani](https://community.temporal.io/u/Kani)\ **Post date:** [September 12, 2022, 1:37pm UTC](https://community.temporal.io/t/history-node-keeps-growing/5944/1 "2022-09-12T13:37:57Z")
I have been playing around with temporal. Created several workflow executions on a namespace with 3 days retention. Workflow executions seems to be purged from the UI after that period of time, but history\_node keeps growing, I don’t see it purged. Workflow executions have local and non-local activities and are completed in less than 1 minute (no long executions or strange timers). Activities’s arguments and return values are complex objects (I know this is not recommended and I should change that), but table size is not stabilizing as expected after 4 days running on stabilized traffic. Tried to build up a query over this table to understand if the purged is being done or not, but I failed. Is there any metric / log that I can check to see if history\_node is being purged correctly? Thanks in advance! ---
**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\ **Post date:** [September 13, 2022, 3:12am UTC](https://community.temporal.io/t/history-node-keeps-growing/5944/2 "2022-09-13T03:12:44Z")
What’s persistence do you configure? `tctl adm cl d | jq .persistenceStore` What’s your set namespace retention? `tctl --ns n desc` Temporal worker service does have a system workflow that removes execution info for closed workflows that have reached namespace retention, however currently it’s only enabled when using Cassandra persistence. Will keep checking for SQL and report back. ---
**Author:** ![Kani](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/kani/32/2504_2.png) [@Kani](https://community.temporal.io/u/Kani)\ **Post date:** [September 13, 2022, 12:32pm UTC](https://community.temporal.io/t/history-node-keeps-growing/5944/3 "2022-09-13T12:32:57Z")
It’s configured a sql datastore, namespace retention was 3 days, but I changed that yesterday to 1 day. > Temporal worker service does have a system workflow that removes execution info for closed workflows that have reached namespace retention, however currently it’s only enabled when using Cassandra persistence. So on SQL datastore, Workflow Execution data remains forever? That’s a huge problem. ---
**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\ **Post date:** [September 13, 2022, 12:53pm UTC](https://community.temporal.io/t/history-node-keeps-growing/5944/4 "2022-09-13T12:53:10Z")
Working with server team to clarify this and will get back to you. ---
**Author:** ![Kani](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/kani/32/2504_2.png) [@Kani](https://community.temporal.io/u/Kani)\ **Post date:** [September 13, 2022, 1:10pm UTC](https://community.temporal.io/t/history-node-keeps-growing/5944/5 "2022-09-13T13:10:39Z")
The version of temporal we are using is 1.16.2 I have made a quick search on temporal code and I found [temporal/scavenger.go at master · temporalio/temporal · GitHub](https://github.com/temporalio/temporal/blob/master/service/worker/scanner/history/scavenger.go), which seems like a cleaner task for history events. I’m not a good Go dev, but the comment here bring some light: [temporal/scavenger.go at master · temporalio/temporal · GitHub](https://github.com/temporalio/temporal/blob/master/service/worker/scanner/history/scavenger.go#L68) ---
**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\ **Post date:** [September 13, 2022, 5:42pm UTC](https://community.temporal.io/t/history-node-keeps-growing/5944/6 "2022-09-13T17:42:08Z")
Was able to get more info on this Yes “garbage history” cleanup does not happen for SQL currently like it does for Cassandra. This is something that will be addressed in future releases. Was told however that the footprint of these “garbage” records that are not removed should be very small (should contain only the first event of the execution history) and it should not cause significant issues in almost all cases. ---
**Author:** ![Kani](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/kani/32/2504_2.png) [@Kani](https://community.temporal.io/u/Kani)\ **Post date:** [September 13, 2022, 5:46pm UTC](https://community.temporal.io/t/history-node-keeps-growing/5944/7 "2022-09-13T17:46:27Z")
Is there anyway I can track those records manually? ---
**Author:** ![dankow](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@dankow](https://community.temporal.io/u/dankow)\ **Post date:** [September 20, 2022, 11:07pm UTC](https://community.temporal.io/t/history-node-keeps-growing/5944/8 "2022-09-20T23:07:09Z")
I’m interested in any updates to this conversation. We use MySQL for persistence and have a history\_node table that’s 1.6 TB and growing steadily. ---
**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\ **Post date:** [September 21, 2022, 3:41am UTC](https://community.temporal.io/t/history-node-keeps-growing/5944/9 "2022-09-21T03:41:52Z")
Thanks for reporting, which server version are you running? ---
**Author:** ![dankow](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@dankow](https://community.temporal.io/u/dankow)\ **Post date:** [September 21, 2022, 4:29am UTC](https://community.temporal.io/t/history-node-keeps-growing/5944/10 "2022-09-21T04:29:50Z")
This cluster is on 1.8. The size of the `history_node` table is one reason we haven’t upgraded past that version. ---
**Author:** ![Kani](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/kani/32/2504_2.png) [@Kani](https://community.temporal.io/u/Kani)\ **Post date:** [September 21, 2022, 12:30pm UTC](https://community.temporal.io/t/history-node-keeps-growing/5944/11 "2022-09-21T12:30:55Z")
We had made some changes on the workflow to avoid storing data not related to the workflow itself, that made an average reduction of 25% of the event-data. By doing so, I got rid of the message saying that the Blob size warning log, too. On the other hand, I reduced the retention days of the namespace drastically. Finally, after applying the below changes, we ended up with 50% of dead-tuples on that table, so we changed some configuration (I don’t remember which ones right now) over the VACCUM strategy for that table and the TOAST associated. The table seems not to be growing so far, VACCUM seems to be triggering normally and the percentage of dead-tuples seems under control; but I am monitoring closely. ---
**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\ **Post date:** [September 21, 2022, 3:06pm UTC](https://community.temporal.io/t/history-node-keeps-growing/5944/12 "2022-09-21T15:06:12Z")
Team opened issue [here](https://github.com/temporalio/temporal/issues/3419) for this to be addressed for SQL persistence. ---
**Author:** ![Ryan\_Qian](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ryan_qian/32/583_2.png) [@Ryan\_Qian](https://community.temporal.io/u/Ryan_Qian)\ **Post date:** [January 16, 2023, 3:11am UTC](https://community.temporal.io/t/history-node-keeps-growing/5944/13 "2023-01-16T03:11:50Z")
which version should I upgrade to if need this scavenger feature? can I use `tctl adm db clean` to cleanup db without upgrade temporal version(we are at 1.17.0 and mysql engine)? @tihomir Or any manually cleanup sql I can used to cleanup history\_node table? then I can upgrade temporal cluster later.