# Cassandra history\_node table keeps growing **URL:** \ **Category:** Community Support\ **Tags:** cassandra\ **Created:** [May 8, 2023, 3:22pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181 "2023-05-08T15:22:33Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![ravikiran](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@ravikiran](https://community.temporal.io/u/ravikiran)\ **Post date:** [May 8, 2023, 3:22pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/1 "2023-05-08T15:22:33Z")
hi, Cassandra history\_node table keeps growing. We have set namespace retention of 7 days. We observe this on both server versions 1.9.2 and 1.20.1 ![Screenshot 2023-05-05 at 9.36.45 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/1/1db43a422cbcac82fad1506f901d2cd39083ff08.png) Did check this related [thread](https://community.temporal.io/t/history-node-keeps-growing/5944/2). Since we have configured, Cassandra as persistence store, we should not have encountered this issue? Or are we missing anything? Appreciate any help. ---
**Author:** ![ravikiran](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@ravikiran](https://community.temporal.io/u/ravikiran)\ **Post date:** [May 8, 2023, 3:37pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/2 "2023-05-08T15:37:48Z")
Output of tctl commands > /etc/temporal $ tctl adm cl d | jq .persistenceStore > “cassandra” > /etc/temporal $ tctl --ns samples-namespace n desc > Name: samples-namespace > Id: 5a369311-26c0-4533-9b1d-8f54d23298b8 > Description: > OwnerEmail: > NamespaceData: map[string]string(nil) > State: Registered > Retention: 168h0m0s > ActiveClusterName: active > Clusters: active > HistoryArchivalState: Disabled > IsGlobalNamespace: false > FailoverVersion: 0 > FailoverHistory: > VisibilityArchivalState: Disabled > Bad binaries to reset: > ±----------------±---------±-----------±-------+ > | BINARY CHECKSUM | OPERATOR | START TIME | REASON | > ±----------------±---------±-----------±-------+ > ±----------------±---------±-----------±-------+ ---
**Author:** ![ravikiran](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@ravikiran](https://community.temporal.io/u/ravikiran)\ **Post date:** [May 9, 2023, 6:03pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/3 "2023-05-09T18:03:10Z")
@tihomir@maxim any suggestions here? ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [May 13, 2023, 4:25pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/5 "2023-05-13T16:25:22Z")
Did it stop growing after 7 days? Is Cassandra’s compaction configured correctly? ---
**Author:** ![ravikiran](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@ravikiran](https://community.temporal.io/u/ravikiran)\ **Post date:** [May 15, 2023, 10:42am UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/6 "2023-05-15T10:42:22Z")
> Did it stop growing after 7 days? No. Its growing for months now. However, workflows older than 7 days are not visible in UI. > Is Cassandra’s compaction configured correctly? We used the Schema creation script present in the github repo to create the tables. `CREATE TABLE temporal.history_node ( tree_id uuid, branch_id uuid, node_id bigint, txn_id bigint, data blob, data_encoding text, prev_txn_id bigint, PRIMARY KEY (tree_id, branch_id, node_id, txn_id) ) WITH CLUSTERING ORDER BY (branch_id ASC, node_id ASC, txn_id DESC) AND additional_write_policy = '99p' AND bloom_filter_fp_chance = 0.1 AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} AND cdc = false AND comment = '' AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'} AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND crc_check_chance = 1.0 AND default_time_to_live = 0 AND extensions = {} AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair = 'BLOCKING' AND speculative_retry = '99p';` ---
**Author:** ![Pradeep](https://avatars.discourse-cdn.com/v4/letter/p/f0a364/32.png) [@Pradeep](https://community.temporal.io/u/Pradeep)\ **Post date:** [May 15, 2023, 12:58pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/7 "2023-05-15T12:58:55Z")
@maxim Is the tree\_id the workflow id? How can we ensure that completed workflows which are crossed the configured retention period is not in history node table? Ideally the completed workflows after retention should be removed from the history node, isn’t? ---
**Author:** ![ravikiran](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@ravikiran](https://community.temporal.io/u/ravikiran)\ **Post date:** [May 15, 2023, 2:10pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/8 "2023-05-15T14:10:02Z")
Additional info: ![image-2023-05-15-17-22-24-230](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/a/afc297b1cd0a194166ba9c4d46d98c47b1d28bd2.png) ![image-2023-05-15-17-20-10-811](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/c/caece1e62fb3e01b9499bec6fe2a614765e4f07e.png) ---
**Author:** ![mindaugas](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/mindaugas/32/3782_2.png) [@mindaugas](https://community.temporal.io/u/mindaugas)\ **Post date:** [May 15, 2023, 6:17pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/9 "2023-05-15T18:17:22Z")
Could you check the compaction configuration? See [Compaction | Apache Cassandra Documentation](https://cassandra.apache.org/doc/latest/cassandra/operating/compaction/index.html) for more information. Also, you can try forcing compaction manually with [nodetool compact](https://docs.datastax.com/en/dse/5.1/dse-dev/datastax_enterprise/tools/nodetool/toolsCompact.html) From the issue description, it sounds like `autocompaction` is not enabled or the `gc_grace_seconds` period is too large. Btw, compaction requires ~2x disk size. So in your case it might require ~6T free disk space. Otherwise compaction might fail. ---
**Author:** ![ravikiran](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@ravikiran](https://community.temporal.io/u/ravikiran)\ **Post date:** [May 16, 2023, 2:15pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/10 "2023-05-16T14:15:09Z")
Compaction is configured and running correctly. There is 3x disk space, so no reason for compaction to fail. How often delete statement is executed against the history\_node table. [temporal/history\_store.go at 52b03657479941f60592163c0f2284a742d0fc84 · temporalio/temporal · GitHub](https://github.com/temporalio/temporal/blob/52b03657479941f60592163c0f2284a742d0fc84/common/persistence/cassandra/history_store.go#L56) Is this part of scavenger workflow? Is there a known schedule for scavenger worfklows? Can it be configurable? ---
**Author:** ![Michael\_Snowden](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/michael_snowden/32/3634_2.png) [@Michael\_Snowden](https://community.temporal.io/u/Michael_Snowden)\ **Post date:** [May 16, 2023, 3:50pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/11 "2023-05-16T15:50:43Z")
What’s your dynamic config value for “worker.historyScannerEnabled”? ---
**Author:** ![ravikiran](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@ravikiran](https://community.temporal.io/u/ravikiran)\ **Post date:** [May 16, 2023, 4:42pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/12 "2023-05-16T16:42:13Z")
We haven’t set this. It is default value. ---
**Author:** ![ravikiran](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@ravikiran](https://community.temporal.io/u/ravikiran)\ **Post date:** [May 17, 2023, 12:22pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/13 "2023-05-17T12:22:34Z")
hi @maxim , We found records in history\_node table which does not have corresponding records in executions view. ![Screenshot 2023-05-17 at 5.42.34 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/b/bc4cc8cdfc42dc9d4e5e9daeb607093eac1a515a.jpeg) ![Screenshot 2023-05-17 at 5.44.10 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/f/f748b42f9cc0a15cfb0a341cdc4d951bdfab2e0d.png) Queries: 1. Since we cannot find execution history for this run\_id, should’nt this record be deleted in history\_node table by scavenger workflow? 2. None of the other tables have crossed 50 GB in size but history\_node seems to have ~7 TB in size. @maxim pls help support and suggest course of action. ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [May 17, 2023, 4:58pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/14 "2023-05-17T16:58:54Z")
Did you use reset command? ---
**Author:** ![ravikiran](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@ravikiran](https://community.temporal.io/u/ravikiran)\ **Post date:** [May 17, 2023, 5:16pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/15 "2023-05-17T17:16:46Z")
no, we did not. ---
**Author:** ![ravikiran](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@ravikiran](https://community.temporal.io/u/ravikiran)\ **Post date:** [May 19, 2023, 5:03am UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/16 "2023-05-19T05:03:22Z")
@maxim is this an issue in Cassandra persistence of Temporal? ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [May 20, 2023, 3:32pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/17 "2023-05-20T15:32:28Z")
I"m not aware of such an issue, as we run many clusters that use Cassandra for persistence. ---
**Author:** ![ravikiran](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@ravikiran](https://community.temporal.io/u/ravikiran)\ **Post date:** [May 22, 2023, 2:38pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/18 "2023-05-22T14:38:23Z")
Could this be the reason? > _Leveled compaction strategy does a better job of trying to keep data for a partition in a limited range of sstables, but if you wrote data some time ago and it’s aged into higher tiers and then come along later and do the delete, it can take a good deal of time for the delete to make it’s way up into the higher leveled tiers. This is the inherent problem with issuing deletes after the fact expecting to free up disk space. Issuing the initial write and any subsequent update with a TTL reduces the issue as the tombstone after the TTL has elapse is the original record as well so you avoid that nasty issue of having to wait for stable 1 to be able to compact with sstable 100_ Our DBA’s are suggesting we alter the compaction’s subproperties especially the tombstone\_threshold from 0.2 to 0.05 ([Compaction subproperties](https://docs.datastax.com/en/cql-oss/3.1/cql/cql_reference/compactSubprop.html)) @maxim pls let us know your thoughts on this. ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [May 22, 2023, 3:57pm UTC](https://community.temporal.io/t/cassandra-history-node-table-keeps-growing/8181/19 "2023-05-22T15:57:28Z")
Unfortunately, I’m not an expert on Cassandra’s compaction strategies.