After upgrading server from V1.18.5 to V1.19.0 , we are seeing below errors in all of the four components
[3] kube.app.logs.gosis-100-frontend-0_89176d63701-temporal-dev.log: [1721644428.534000000, {}, {“level”=>“error”, “ts”=>“2024-07-22T10:33:48.534Z”, “@timestamp”=>“2024-07-22T10:33:48.534Z”, “msg”=>“start failed, rolling back”, “component”=>“fx”, “error”=>“context deadline exceeded”, “logging-call-at”=>“fx.go:1030”, “stacktrace”=>“go.temporal.io/server/common/log.(*zapLogger).Error
/jenkins/workspace/eton-repo-and-configs-to-support/temporal/common/log/zap_logger.go:144
go.temporal.io/server/temporal.(*fxLogAdapter).LogEvent
/jenkins/workspace/eton-repo-and-configs-to-support/temporal/temporal/fx.go:1030
go.uber.org/fx.(*App).start
/jenkins/workspace/eton-repo-and-configs-to-support/temporal/vendor/go.uber.org/fx/app.go:681
go.uber.org/fx.withTimeout.func1
/jenkins/workspace/eton-repo-and-configs-to-support/temporal/vendor/go.uber.org/fx/app.go:784”, “sealId”=>“XXXXX”, “env”=>“XXX”, “kubernetesNamespace”=>“XXXXXXXXX-temporal-dev”, “serverId”=>“100”, “kubernetesCluster”=>“XXXXXXXXXX”, “metaMarker”=>“marker21”}]
[4] kube.app.logs.gosis-100-frontend-0_XXXXXXXXX-temporal-dev.log: [1721644428.534000000, {}, {“level”=>“error”, “ts”=>“2024-07-22T10:33:48.534Z”, “@timestamp”=>“2024-07-22T10:33:48.534Z”, “msg”=>“rollback failed”, “component”=>“fx”, “error”=>“context deadline exceeded”, “logging-call-at”=>“fx.go:1033”, “stacktrace”=>"go.temporal.io/server/common/log.(*zapLogger).Error
/jenkins/workspace/eton-repo-and-configs-to-support/temporal/common/log/zap_logger.go:144
go.temporal.io/server/temporal.(*fxLogAdapter).LogEvent
/jenkins/workspace/eton-repo-and-configs-to-support/temporal/temporal/fx.go:1033
go.uber.org/fx.(*App).start
/jenkins/workspace/eton-repo-and-configs-to-support/temporal/vendor/go.uber.org/fx/app.go:684
go.uber.org/fx.withTimeout.func1
{“level”:“error”,“ts”:“2024-07-22T10:32:30.643Z”,“msg”:“start failed”,“component”:“fx”,“error”:“OnStart hook added by go.temporal.io/server/common/resource.MembershipMonitorProvider failed: context deadline exceeded\n\ngo.temporal.io/server/common/namespace.RegistryLifetimeHooks.func1() took 108.43686ms from:\n\tgo.temporal.io/server/common/namespace.RegistryLifetimeHooks (/jenkins/workspace/eton-repo-and-configs-to-support/temporal/common/namespace/fx.go:41)\ngo.temporal.io/server/common/cluster.MetadataLifetimeHooks.func1() took 56.170859ms from:\n\tgo.temporal.io/server/common/cluster.MetadataLifetimeHooks (/jenkins/workspace/eton-repo-and-configs-to-support/temporal/common/cluster/fx.go:48)\ngo.temporal.io/server/common/metrics.RuntimeMetricsReporterLifetimeHooks.func1() took 407.37µs from:\n\tgo.temporal.io/server/common/metrics.RuntimeMetricsReporterLifetimeHooks (/jenkins/workspace/eton-repo-and-configs-to-support/temporal/common/metrics/fx.go:44)\n”,“logging-call-at”:“fx.go:1037”,“stacktrace”:“go.temporal.io/server/common/log.(*zapLogger).Error\n\t/jenkins/workspace/eton-repo-and-configs-to-support/temporal/common/log/zap_logger.go:144\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/jenkins/workspace/eton-repo-and-configs-to-support/temporal/temporal/fx.go:1037\ngo.uber.org/fx.(*App).Start.func1\n\t/jenkins/workspace/eton-repo-and-configs-to-support/temporal/vendor/go.uber.org/fx/app.go:662\ngo.uber.org/fx.(*App).Start\n\t/jenkins/workspace/eton-repo-and-configs-to-support/temporal/vendor/go.uber.org/fx/app.go:670\ngo.temporal.io/server/temporal.(*ServerImpl).Start.func1\n\t/jenkins/workspace/eton-repo-and-configs-to-support/temporal/temporal/server_impl.go:116”}
{“level”:“info”,“ts”:“2024-07-22T10:33:48.534Z”,“msg”:“Current reachable members”,“service”:“frontend”,“component”:“service-resolver”,“service”:“history”,“addresses”:[“1.1.1.1:XXXX”],“logging-call-at”:“rpServiceResolver.go:283”}
{“level”:“info”,“ts”:“2024-07-22T10:33:48.534Z”,“msg”:“Current reachable members”,“service”:“frontend”,“component”:“service-resolver”,“service”:“worker”,“addresses”:[“1.1.3.1:XXXX”],“logging-call-at”:“rpServiceResolver.go:283”}
{“level”:“info”,“ts”:“2024-07-22T10:33:48.534Z”,“msg”:“Current reachable members”,“service”:“frontend”,“component”:“service-resolver”,“service”:“frontend”,“addresses”:[“1.1.0.1:XXXX”],“logging-call-at”:“rpServiceResolver.go:283”}
{“level”:“info”,“ts”:“2024-07-22T10:33:48.534Z”,“msg”:“Current reachable members”,“service”:“frontend”,“component”:“service-resolver”,“service”:“matching”,“addresses”:[“1.1.2.1:XXXX”],“logging-call-at”:“rpServiceResolver.go:283”}
{“level”:“error”,“ts”:“2024-07-22T10:33:48.534Z”,“msg”:“start failed, rolling back”,“component”:“fx”,“error”:“context deadline exceeded”,“logging-call-at”:“fx.go:1030”,“stacktrace”:“go.temporal.io/server/common/log.(*zapLogger).Error\n\t/jenkins/workspace/eton-repo-and-configs-to-support/temporal/common/log/zap_logger.go:144\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/jenkins/workspace/eton-repo-and-configs-to-support/temporal/temporal/fx.go:1030\ngo.uber.org/fx.(*App).start\n\t/jenkins/workspace/eton-repo-and-configs-to-support/temporal/vendor/go.uber.org/fx/app.go:681\ngo.uber.org/fx.withTimeout.func1\n\t/jenkins/workspace/eton-repo-and-configs-to-support/temporal/vendor/go.uber.org/fx/app.go:784”}
{“level”:“error”,“ts”:“2024-07-22T10:33:48.534Z”,“msg”:“rollback failed”,“component”:“fx”,“error”:“context deadline exceeded”,“logging-call-at”:“fx.go:1033”,“stacktrace”:“go.temporal.io/server/common/log.(*zapLogger).Error\n\t/jenkins/workspace/eton-repo-and-configs-to-support/temporal/common/log/zap_logger.go:144\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/jenkins/workspace/eton-repo-and-configs-to-support/temporal/temporal/fx.go:1033\ngo.uber.org/fx.(*App).start\n\t/jenkins/workspace/eton-repo-and-configs-to-support/temporal/vendor/go.uber.org/fx/app.go:684\ngo.uber.org/fx.withTimeout.func1\n\t/jenkins/workspace/eton-repo-and-configs-to-support/temporal/vendor/go.uber.org/fx/app.go:784”}
Any help is appreciated