I deploy the temporal cluster in k8s, and each service has two instances. When I try to test the concurrent performance by maru, I encountered an error.
The frontend service logs is:
{"level":"info","ts":"2022-12-03T02:33:52.531Z","msg":"matching client encountered error","service":"frontend","error":"service rate limit exceeded","service-error-type":"serviceerror.ResourceExhausted","logging-call-at":"metric_client.go:217"}
The history service logs is:
{"level":"error","ts":"2022-12-05T02:27:29.087Z","msg":"Fail to process task","shard-id":461,"address":"10.244.1.112:17234","component":"visibility-queue-processor","wf-namespace-id":"2dc8a5ed-aa72-4801-8045-16b95767bd56","wf-id":"basic-workflow-2e24fcdf-c3e9-4629-8c16-21205d1e6f46-0-184-1","wf-run-id":"64d8fb96-9b04-4d74-9baf-7bfd21cbc8a9","queue-task-id":5243799,"queue-task-visibility-timestamp":"2022-12-05T02:27:24.182Z","queue-task-type":"VisibilityStartExecution","queue-task":{"NamespaceID":"2dc8a5ed-aa72-4801-8045-16b95767bd56","WorkflowID":"basic-workflow-2e24fcdf-c3e9-4629-8c16-21205d1e6f46-0-184-1","RunID":"64d8fb96-9b04-4d74-9baf-7bfd21cbc8a9","VisibilityTimestamp":"2022-12-05T02:27:24.182222572Z","TaskID":5243799,"Version":0},"wf-history-event-id":0,"error":"shard status unknown","lifecycle":"ProcessingFailed","logging-call-at":"lazy_logger.go:68","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/log.(*lazyLogger).Error\n\t/home/builder/temporal/common/log/lazy_logger.go:68\ngo.temporal.io/server/service/history/queues.(*executableImpl).HandleErr\n\t/home/builder/temporal/service/history/queues/executable.go:289\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:226\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:170\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:194\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:171\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:235\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:211"}
{"level":"error","ts":"2022-12-05T02:27:29.087Z","msg":"Operation failed with internal error.","error":"GetVisibilityTasks operation failed. Select failed. Error: context deadline exceeded","metric-scope":20,"logging-call-at":"persistenceMetricClients.go:1461","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/persistence.(*metricEmitter).updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1461\ngo.temporal.io/server/common/persistence.(*executionPersistenceClient).GetHistoryTasks\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:439\ngo.temporal.io/server/common/persistence.(*executionRetryablePersistenceClient).GetHistoryTasks.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:366\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:194\ngo.temporal.io/server/common/persistence.(*executionRetryablePersistenceClient).GetHistoryTasks\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:370\ngo.temporal.io/server/service/history.(*visibilityQueueProcessorImpl).readTasks\n\t/home/builder/temporal/service/history/visibilityQueueProcessor.go:336\ngo.temporal.io/server/service/history.(*queueAckMgrImpl).readQueueTasks\n\t/home/builder/temporal/service/history/queueAckMgr.go:123\ngo.temporal.io/server/service/history.(*queueProcessorBase).processBatch\n\t/home/builder/temporal/service/history/queueProcessorBase.go:237\ngo.temporal.io/server/service/history.(*queueProcessorBase).processorPump\n\t/home/builder/temporal/service/history/queueProcessorBase.go:196"}
{"level":"error","ts":"2022-12-05T02:27:29.087Z","msg":"Processor unable to retrieve tasks","shard-id":415,"address":"10.244.1.112:17234","component":"visibility-queue-processor","error":"context deadline exceeded","logging-call-at":"queueProcessorBase.go:239","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_l
What is causing this problem, and how do I solve it?