We are trying to deploy Temporal using the helm chart, using a MySQL as its persistence store. We’ve successfully deployed the helm chart both in development and on an AWS cluster but on an Azure cluster we are seeing the following error, on all services, leading all pods to go into a crashloopbackoff loop.
{"level":"warn","ts":"2022-08-22T23:36:08.421Z","msg":"Failed to save cluster metadata.","component":"metadata-initializer","error":"proto: ClusterMetadata: wiretype end group for non-group","cluster-name":"active","logging-call-at":"fx.go:628"}
Unable to start server. Error: could not build arguments for function "go.temporal.io/server/temporal".ServerLifetimeHooks (/home/builder/temporal/temporal/fx.go:738): failed to build temporal.Server: could not build arguments for function "go.temporal.io/server/temporal".glob..func1 (/home/builder/temporal/temporal/server_impl.go:65): failed to build *temporal.ServerImpl: could not build arguments for function "go.temporal.io/server/temporal".NewServerFxImpl (/home/builder/temporal/temporal/server_impl.go:69): could not build value group *temporal.ServicesMetadata[group="services"]: could not build arguments for function "go.temporal.io/server/temporal".HistoryServiceProvider (/home/builder/temporal/temporal/fx.go:334): failed to build config.Persistence: received non-nil error from function "go.temporal.io/server/temporal".ApplyClusterMetadataConfigProvider (/home/builder/temporal/temporal/fx.go:563): error while fetching cluster metadata: proto: ClusterMetadata: wiretype end group for non-group
Given that the same helm chart works on one kubernetes environment and not another, I’m guessing there must be some configuration issue though it’s hard to tell from the error what configuration that might be.