SIGSEGV in temporal-history service

Community Support
,
1

Getting below error in temporal-history service, rest services are running fine.

{“level”:“info”,“ts”:“2024-01-23T12:32:48.434Z”,“msg”:“none”,“shard-id”:2000,“address”:“172.22.0.9:7234”,“component”:“transfer-queue-processor”,“lifecycle”:“Started”,“logging-call-at”:“queue_immediate.go:123”}
{“level”:“info”,“ts”:“2024-01-23T12:32:48.435Z”,“msg”:“none”,“shard-id”:2000,“address”:“172.22.0.9:7234”,“component”:“visibility-queue-processor”,“lifecycle”:“Starting”,“logging-call-at”:“queue_immediate.go:114”}
{“level”:“info”,“ts”:“2024-01-23T12:32:48.434Z”,“msg”:“queue reader started”,“shard-id”:2000,“address”:“172.22.0.9:7234”,“component”:“transfer-queue-processor”,“queue-reader-id”:0,“lifecycle”:“Started”,“logging-call-at”:“reader.go:182”}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1e4aae3]

goroutine 774 [running]:
go.temporal.io/server/common/namespace.(*Namespace).Name(...)
/home/builder/temporal/common/namespace/namespace.go:169
go.temporal.io/server/service/history/queues.(*executableImpl).Execute(0xc004b043c0)
/home/builder/temporal/service/history/queues/executable.go:188 +0x1a3
go.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1()
/home/builder/temporal/common/tasks/fifo_scheduler.go:223 +0x37
go.temporal.io/server/common/backoff.ThrottleRetry.func1({0x60, 0xc000601000})
/home/builder/temporal/common/backoff/retry.go:119 +0x1b
go.temporal.io/server/common/backoff.ThrottleRetryContext({0x2be0260, 0xc0001a2000}, 0xc0034f6e88, {0x2bc5e40, 0x3f6fe60}, 0x0?)
/home/builder/temporal/common/backoff/retry.go:145 +0x20a
go.temporal.io/server/common/backoff.ThrottleRetry(0x2bfca28?, {0x2bc5e40?, 0x3f6fe60?}, 0xc00122ac00?)
/home/builder/temporal/common/backoff/retry.go:120 +0x59
go.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask(0x2bfd5c0, {0x2bfca28, 0xc004b043c0})
/home/builder/temporal/common/tasks/fifo_scheduler.go:233 +0xbf
go.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask(0x2bfd5c0, 0xc0012ec600)
/home/builder/temporal/common/tasks/fifo_scheduler.go:211 +0xa6
created by go.temporal.io/server/common/tasks.(*FIFOScheduler[...]).startWorkers
/home/builder/temporal/common/tasks/fifo_scheduler.go:178 +0x30

Server details:
Database store: MySQL 8.0
|Temporal Server Version |1.21.0|
|Temporal UI Version |2.16.2|

2

Can you give more info, are there any more errors you see in history service? Can you check service_errors server metric:

sum(rate(service_error_with_type{service_type="history"}[5m])) by (error_type)

is the error intermittent maybe happens during/after service restarts?

3

  1. unable to see service_type=history in grafana as container restarting continously due to
    SIGSEGV.

    52aa7ceceb20 temporalio/server:1.21.0 “/etc/temporal/entry…” 5 days ago Restarting (2) 49 seconds ago temporal-history

  2. also, we are seeing this error from logs in history
    {
    “level”: “error”,
    “ts”: “2024-01-29T06:08:19.016Z”,
    “msg”: “Unable to process new range”,
    “shard-id”: 1649,
    “address”: “172.22.0.8:7234”,
    “component”: “timer-queue-processor”,
    “error”: “shard status unknown”,
    “logging-call-at”: “queue_base.go:316”,
    “stacktrace”: “go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/service/history/queues.(*queueBase).processNewRange\n\t/home/builder/temporal/service/history/queues/queue_base.go:316\ngo.temporal.io/server/service/history/queues.(*scheduledQueue).processEventLoop\n\t/home/builder/temporal/service/history/queues/queue_scheduled.go:218”
    }

is the error intermittent maybe happens during/after service restarts?
→ service was running in intial setup and workflows were also created, but after restart we faced this issue.