Error: Task queue range ID was X when it was should have been Y

Support
#1

Question:

What is the severity of this error log message and is the issue automatically reconciled?

{
  "level": "error",
  "ts": "2022-03-22T00:23:20.525Z",
  "msg": "Persistent store operation failure",
  "service": "matching",
  "component": "matching-engine",
  "wf-task-queue-name": "/_sys/app-name/3",
  "wf-task-queue-type": "Activity",
  "wf-namespace": "default",
  "store-operation": "update-task-queue",
  "error": "Task queue range ID was 631 when it was should have been 630",
  "logging-call-at": "taskReader.go:187",
  "stacktrace": "go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:187\ngo.temporal.io/server/internal/goro.(*Group).Go.func1\n\t/temporal/internal/goro/group.go:57"
}

Story:

I am testing a database outage to understand how Temporal behaves in the scenario. A bunch of these errors were logged over the course of 8 hours after the event.

How to reproduce:

Remove the database credentials that the server components use to connect to the database for approximately 5 minutes. I am using mysql. Restore the credential or update the components with new credentials that do have access.

Source

#2

Hi @buhay

What is the severity of this error log message and is the issue automatically reconciled?

If the error is transient, meaning it resolves itself then it can be ignored.

#3

By transient or resolved, I take you to mean that log lines identifying the mismatch in task queue range id eventually stop? Or should I be looking for another log line that identifies the mismatch as reconciled?

#4

that log lines identifying the mismatch in task queue range id eventually stop

Yes, in your described outage test you are blocking and then enabling DB access it seems and these errors can be expected in those cases. Just make sure that they are transient (eventually stop after enabling access, as you mention).

1 Like