Question:
What is the severity of this error log message and is the issue automatically reconciled?
{
"level": "error",
"ts": "2022-03-22T00:23:20.525Z",
"msg": "Persistent store operation failure",
"service": "matching",
"component": "matching-engine",
"wf-task-queue-name": "/_sys/app-name/3",
"wf-task-queue-type": "Activity",
"wf-namespace": "default",
"store-operation": "update-task-queue",
"error": "Task queue range ID was 631 when it was should have been 630",
"logging-call-at": "taskReader.go:187",
"stacktrace": "go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:187\ngo.temporal.io/server/internal/goro.(*Group).Go.func1\n\t/temporal/internal/goro/group.go:57"
}
Story:
I am testing a database outage to understand how Temporal behaves in the scenario. A bunch of these errors were logged over the course of 8 hours after the event.
How to reproduce:
Remove the database credentials that the server components use to connect to the database for approximately 5 minutes. I am using mysql. Restore the credential or update the components with new credentials that do have access.