High volume of context deadline exceeded from visiblity_manager_metrics.go

Community Support
1

Hi,
I have a high volume of these errors in the temporal server logs. I’m using Temporal 1.22.2 with Visibility on Mysql. It’s around ~100 logs / sec so basically my temporal log is completely dominated by these. Does anyone have any suggestions on what I could be looking at the fix this?

{
    "level": "error",
    "ts": "2024-02-13T01:18:48.686Z",
    "msg": "Operation failed with an error.",
    "error": "context deadline exceeded",
    "logging-call-at": "visiblity_manager_metrics.go:264",
    "stacktrace": "go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/temporal/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/persistence/visibility.(*visibilityManagerMetrics).updateErrorMetric\n\t/home/runner/work/temporal/temporal/common/persistence/visibility/visiblity_manager_metrics.go:264\ngo.temporal.io/server/common/persistence/visibility.(*visibilityManagerMetrics).UpsertWorkflowExecution\n\t/home/runner/work/temporal/temporal/common/persistence/visibility/visiblity_manager_metrics.go:118\ngo.temporal.io/server/service/history.(*visibilityQueueTaskExecutor).upsertExecution\n\t/home/runner/work/temporal/temporal/service/history/visibility_queue_task_executor.go:333\ngo.temporal.io/server/service/history.(*visibilityQueueTaskExecutor).processUpsertExecution\n\t/home/runner/work/temporal/temporal/service/history/visibility_queue_task_executor.go:234\ngo.temporal.io/server/service/history.(*visibilityQueueTaskExecutor).Execute\n\t/home/runner/work/temporal/temporal/service/history/visibility_queue_task_executor.go:118\ngo.temporal.io/server/service/history/queues.(*executableImpl).Execute\n\t/home/runner/work/temporal/temporal/service/history/queues/executable.go:236\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/runner/work/temporal/temporal/common/tasks/fifo_scheduler.go:223\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/runner/work/temporal/temporal/common/backoff/retry.go:119\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/runner/work/temporal/temporal/common/backoff/retry.go:145\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/runner/work/temporal/temporal/common/backoff/retry.go:120\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/runner/work/temporal/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/runner/work/temporal/temporal/common/tasks/fifo_scheduler.go:211"
}