Temporal version 1.22.4: Unable to start server. Error: sql schema version compatibility check failed

Hi all,
I’m having an issue to startup the k8s environment with temporal 1.22.4.

Description:

I’m encountering an issue while provisioning the Temporal server (version 1.22.4) on a k3d cluster locally. The Temporal server fails to start due to a SQL schema version compatibility check error.
I followed the steps mentioned here GitHub - temporalio/helm-charts: Temporal Helm charts is the config I am using for postgres: helm-charts/charts/temporal/values/values.postgresql.yaml at main · temporalio/helm-charts · GitHub
The detailed error message is as follows:

sql schema version compatibility check failed: pq: pg_hba.conf rejects connection for host "10.42.0.21", user "postgres", database "temporal", SSL off
Unable to create server. Error: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2 (/go/pkg/mod/go.uber.org/fx@v1.20.0/module.go:251): failed to build fxevent.Logger: could not build arguments for function "go.temporal.io/server/temporal".glob..func8 (/home/builder/temporal/temporal/fx.go:1037): failed to build log.Logger: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider (/home/builder/temporal/temporal/fx.go:173): sql schema version compatibility check failed: pq: pg_hba.conf rejects connection for host "10.42.0.21", user "postgres", database "temporal", SSL off.

Environment Details:

  • Temporal Version: 1.22.4
  • Deployment: k3d cluster (local)
  • Database: PostgreSQL

Issue

The error suggests that the connection to the PostgreSQL database is being rejected due to SSL being off. It appears that the Temporal server is unable to perform the SQL schema version compatibility check because the pg_hba.conf configuration on the PostgreSQL database is rejecting the connection.

I am using the recommended version only. So I have installed the zalando postgres operator. And deployed a 3 instance postgres cluster with the below config:

apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
  name: temporal-backend
  namespace: temporal
spec:
  teamId: "temporal"
  volume:
    size: 10Gi
  numberOfInstances: 3
  users:
    temporal_admin:
    - superuser
    - createdb
  databases:
    temporal: temporal_admin
    temporal_visibility: temporal_admin
  postgresql:
    version: "12"

Later once it was up, I initialized it with the below script and tables got created:

# in https://github.com/temporalio/temporal git repo dir
export SQL_PLUGIN=postgres12
export SQL_HOST=postgresql_host
export SQL_PORT=5432
export SQL_USER=postgresql_user
export SQL_PASSWORD=postgresql_password

make temporal-sql-tool

./temporal-sql-tool --database temporal create-database
SQL_DATABASE=temporal ./temporal-sql-tool setup-schema -v 0.0
SQL_DATABASE=temporal ./temporal-sql-tool update -schema-dir schema/postgresql/v12/temporal/versioned

./temporal-sql-tool --database temporal_visibility create-database
SQL_DATABASE=temporal_visibility ./temporal-sql-tool setup-schema -v 0.0
SQL_DATABASE=temporal_visibility ./temporal-sql-tool update -schema-dir schema/postgresql/v12/visibility/versioned

Steps to Reproduce:

  1. Deploy a Temporal server (version 1.22.4) on a k3d cluster locally.
  2. Configure the Temporal server to connect to a PostgreSQL database.
  3. Attempt to start the Temporal server.

Expected Behavior:

The Temporal server should connect to the PostgreSQL database, perform the schema compatibility check, and start successfully.

Actual Behavior:

The server fails to start due to the connection being rejected by the PostgreSQL pg_hba.conf configuration when SSL is off.

Additional Information:

It seems like the Temporal server may require SSL for the PostgreSQL connection, or there may be a misconfiguration in the PostgreSQL pg_hba.conf file. Any guidance on how to resolve this issue would be greatly appreciated.

Great thanks for your support
Regards
Ujala Singh