Remote sync match failed and Matching host rps exceeded

Hello.

I have an application running with Cadence, and everything seems to be working as expected. All the workflows that i start, finishes. But, when we receive some heavy load some errors starts to appear in cadence-matching and cadence-history.

{"component": "matching-engine","service": "cadence-matching","error": "RemoteSyncMatchedError{Message: remote sync match failed}","msg": "Failed to add task"}
{"wf-domain-id":"8f8e71f8-aa1e-4b8f-8d30-97a68db6f1b2","queue-task-id":4194487,"level":"error","xdc-failover-version":-24,"shard-id":3351,"logging-call-at":"task.go:308","queue-task-type":0,"wf-run-id":"3399e341-7d2b-4d1f-bebd-3590f227ed24","error":"ServiceBusyError{Message: Matching host rps exceeded}","queue-task-visibility-timestamp":1.621959880275e+18,"service":"cadence-history","component":"transfer-queue-processor","wf-id":"f13c5bee-3e9c-45bf-9a9c-2a35e0684622JourneyWorkflow","address":"10.52.16.5:7934","cluster-name":"active","ts":"2021-05-25T16: 24: 42.951Z","lifecycle":"ProcessingFailed","msg":"Fail to process task"}

Is it something that we should be concerned with? :thinking: Can we increase some config to avoid these errors? We still have hadware if it needs.

Reading the docs more calmly, i found MatchingRPS and MatchingPersistenceMaxQPS configs. Is it ok to increase these numbers?

In my understanding it will increase the load in cassandra, right?