i see my frontend filled with these logs {"level":"error","ts":"2022-03-14T07:20:31.078Z","msg":"history client encountered error","service":"frontend","error":"sql: no rows in result set","service-error-type":"serviceerror.NotFound","logging-call-at":"metricClient.go:620","stacktrace":"go.temporal…

[image] tihomir: if that’s o Thanks @tihomir Between i see my replication taks and history node has too much data check this out SELECT -> TABLE_NAME AS `Table`, -> ROUND((DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024) AS `Size (MB)` -> FROM -> information_schema.TABLES …

replication_tasks stores replication tasks which is used for cross dc replication. If your namespace is configured to be replicate to multiple clusters, but the remote cluster never come to fetch the replication tasks, then the replication tasks will not be cleanup. history_node stores all your wor…

[image] Yimin_Chen: sible that the could accumulate that much of data. The thing is i see the replication happening and my remote has exact copy of whats available in primary… thats what make me scratch my head! Between, if i keep swithching between primary and standby… and at some point if …

could you do tctl --ns your_namespace n desc to see how many clusters are configured for your namespace? Is it possible that you have more than 2 clusters configured?

Usually, the replication_tasks should be small as it only stores metadata. Could you check if the metrics has any datapoint? Operation: ReplicationTaskCleanup name: replication_task_cleanup_count Secondly, Could you pick a shard and run tctl admin shard describe --shard_id?

[image] yux: Secondly, Could you pick a shard and run tctl admin shard describe --shard_id ? this shows as tctl admin shard describe --shard_id 278 { "shardId": 278, "rangeId": "317", "owner": "10.xx.yyy.zz:7234", "transferAckLevel": "332398619", "updateTime": "2022-03-16T17:37:10.…

[image] yux: Operation: ReplicationTaskCleanup name: replication_task_cleanup_count current cleanup count is 936. |replication_task_cleanup_count{app_kubernetes_io_component="history",app_kubernetes_io_instance="xxxal",app_kubernetes_io_managed_by="Helm",app_kubernetes_io_name="temporal",app…

historically too i see cleanup count in order of 400 to 1500 [image]

tctl --ns myns-dev n desc Name: myns-dev Id: ff24a0e9-f4eb-4590-8c52-0c560dc87155 Description: my temporal namespace OwnerEmail: me@mydomain.com NamespaceData: map[string]string(nil) State: Registered Retention: 24h0m0s ActiveClusterName: active Clusters: active, standby HistoryArchivalState: Enabl…

i just tried cleaning up my standby cluster and allowed to replicate data afresh… i see the replication is too slow (how ever i do not see any connection drop/network issues etc)… this time around only 3 of my workflows got replicated to standby cluster… i event tried sending some fake signals to…

Errors in frontend and hisotry

Community Support

madhu March 17, 2022, 4:26pm 19

could it be related to Adding a cluster using dns fails - #2 by yux , there dns:/// is breaking and hence no replication happening and over a period replication db queue/table grows and causes db cpu to go high??

Topic		Replies	Views
Whats optimal settings for quick xdc replication? what metrics can one track? Community Support xdc	12	1368	July 8, 2024
CPU Utilization with AWS RDS (MySQL) Community Support mysql	34	3894	March 14, 2022
Temporal production deployment stopped working Community Support java-sdk , helm	7	1258	January 15, 2023
Temporal History Server Errors Community Support history	10	1716	September 4, 2024
Failed to retrieve replication message and Failed to get replication tasks from client errors Community Support replication	6	894	July 8, 2024

Errors in frontend and hisotry

Related topics