Hi team,
We have been facing the below error after creating the workflow which was executed successfully but not visible in UI, and the below error log occurred:
{"level":"error","ts":"2024-01-10T06:30:15.623Z","msg":"Operation failed with an error.","error":"pq: syntax error in tsvector: \"Mocked-TraceId:5c060ad0-5ec8-4ebb-b344-5558a9f11111\"","logging-call-at":"visiblity_manager_metrics.go:264","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/persistence/visibility.(*visibilityManagerMetrics).updateErrorMetric\n\t/home/builder/temporal/common/persistence/visibility/visiblity_manager_metrics.go:264\ngo.temporal.io/server/common/persistence/visibility.(*visibilityManagerMetrics).RecordWorkflowExecutionClosed\n\t/home/builder/temporal/common/persistence/visibility/visiblity_manager_metrics.go:108\ngo.temporal.io/server/service/history.(*visibilityQueueTaskExecutor).recordCloseExecution\n\t/home/builder/temporal/service/history/visibility_queue_task_executor.go:443\ngo.temporal.io/server/service/history.(*visibilityQueueTaskExecutor).processCloseExecution\n\t/home/builder/temporal/service/history/visibility_queue_task_executor.go:392\ngo.temporal.io/server/service/history.(*visibilityQueueTaskExecutor).Execute\n\t/home/builder/temporal/service/history/visibility_queue_task_executor.go:120\ngo.temporal.io/server/service/history/queues.(*executableImpl).Execute\n\t/home/builder/temporal/service/history/queues/executable.go:236\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:223\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:119\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:145\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:120\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:211"}
{"level":"error","ts":"2024-01-10T06:30:15.623Z","msg":"Fail to process task","shard-id":3,"address":"10.123.12.151:7234","component":"visibility-queue-processor","wf-namespace-id":"32a7712b-7885-4ed3-a6a0-7aebe5a661ca","wf-id":"675c2aee-e95f-4ca5-ba94-24f4370fd027","wf-run-id":"7bf44381-2431-443e-a7f5-8366b6f66ff0","queue-task-id":22020493,"queue-task-visibility-timestamp":"2024-01-10T06:18:39.570Z","queue-task-type":"VisibilityCloseExecution","queue-task":{"NamespaceID":"32a7712b-7885-4ed3-a6a0-7aebe5a661ca","WorkflowID":"675c2aee-e95f-4ca5-ba94-24f4370fd027","RunID":"7bf44381-2431-443e-a7f5-8366b6f66ff0","VisibilityTimestamp":"2024-01-10T06:18:39.570108633Z","TaskID":22020493,"Version":0},"wf-history-event-id":0,"error":"pq: syntax error in tsvector: \"Mocked-TraceId:5c060ad0-5ec8-4ebb-b344-5558a9f11111\"","lifecycle":"ProcessingFailed","logging-call-at":"lazy_logger.go:68","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/log.(*lazyLogger).Error\n\t/home/builder/temporal/common/log/lazy_logger.go:68\ngo.temporal.io/server/service/history/queues.(*executableImpl).HandleErr\n\t/home/builder/temporal/service/history/queues/executable.go:347\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:224\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:119\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:145\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:120\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:211"}
{"level":"error","ts":"2024-01-10T06:30:15.623Z","msg":"Critical error processing task, retrying.","shard-id":3,"address":"10.123.12.151:7234","component":"visibility-queue-processor","wf-namespace-id":"32a7712b-7885-4ed3-a6a0-7aebe5a661ca","wf-id":"675c2aee-e95f-4ca5-ba94-24f4370fd027","wf-run-id":"7bf44381-2431-443e-a7f5-8366b6f66ff0","queue-task-id":22020493,"queue-task-visibility-timestamp":"2024-01-10T06:18:39.570Z","queue-task-type":"VisibilityCloseExecution","queue-task":{"NamespaceID":"32a7712b-7885-4ed3-a6a0-7aebe5a661ca","WorkflowID":"675c2aee-e95f-4ca5-ba94-24f4370fd027","RunID":"7bf44381-2431-443e-a7f5-8366b6f66ff0","VisibilityTimestamp":"2024-01-10T06:18:39.570108633Z","TaskID":22020493,"Version":0},"wf-history-event-id":0,"attempt":47,"error":"pq: syntax error in tsvector: \"Mocked-TraceId:5c060ad0-5ec8-4ebb-b344-5558a9f11111\"","operation-result":"OperationCritical","logging-call-at":"lazy_logger.go:68","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/log.(*lazyLogger).Error\n\t/home/builder/temporal/common/log/lazy_logger.go:68\ngo.temporal.io/server/service/history/queues.(*executableImpl).HandleErr.func1\n\t/home/builder/temporal/service/history/queues/executable.go:268\ngo.temporal.io/server/service/history/queues.(*executableImpl).HandleErr\n\t/home/builder/temporal/service/history/queues/executable.go:348\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:224\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:119\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:145\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:120\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:211"}
On further investigation, it seems like the error is due to using the colon ‘:’ character in TraceId registered as a custom attribute.
Could someone please help in how to recover the workflow and make it visible in UI if such an error occurs?