Cannot connect to RDS - `sql schema version compatibility check failed: unable to read DB schema version keyspace/database: temporal error: no usable database connection found`

Hello! I’m trying to connect to use RDS as temporal database, but it crashes on

sql schema version compatibility check failed: unable to read DB schema version keyspace/database: temporal error: no usable database connection found

error. Temporal correctly connects to DB at first, checks migrations and eventually executes them. DB is created, migrations are performed, tables exist in DB. But then this error occurs and I have no idea why as it’s able to connect to execute the migrations.

I’m using Docker compose which works correctly with “local” PG, but not with RDS. this my Docker compose file:

  temporal:
    container_name: temporal
    restart: unless-stopped
    environment:
      - DB=postgres12
      - DB_PORT=5432
      - POSTGRES_SEEDS=***.eu-west-1.rds.amazonaws.com
      - POSTGRES_USER=postgres
      - POSTGRES_PWD=***
      - POSTGRES_TLS_ENABLED=true
      - POSTGRES_TLS_DISABLE_HOST_VERIFICATION=true
      - DYNAMIC_CONFIG_FILE_PATH=config/dynamicconfig/development-sql.yaml
    image: temporalio/auto-setup:1.27@sha256:501a883b1c4a4168eb5000c368495e7eb7106dbadefc75b7fd886be0108456b9
    ports:
      - 7233:7233
    volumes:
      - ./volumes/temporal-dynamicconfig:/etc/temporal/config/dynamicconfig
    extra_hosts:
      - "host.docker.internal:host-gateway"
    labels:
      kompose.volume.type: configMap

  temporal-admin-tools:
    container_name: temporal-admin-tools
    restart: unless-stopped
    depends_on:
      - temporal
    environment:
      - TEMPORAL_ADDRESS=temporal:7233
      - TEMPORAL_CLI_ADDRESS=temporal:7233
      - TEMPORAL_CLI_SHOW_STACKS=1
    image: temporalio/admin-tools:1.25.2-tctl-1.18.1-cli-1.1.1@sha256:da0c7a7982b571857173ab8f058e7f139b3054800abb4dcb100445d29a563ee8
    stdin_open: true
    tty: true
    entrypoint: /etc/temporal/entrypoint.sh
    extra_hosts:
      - "host.docker.internal:host-gateway"
    healthcheck:
      test: ["CMD", "tctl", "workflow", "list"]
      interval: 1s
      timeout: 5s
      retries: 30
    volumes:
      - ./scripts/mirror-name-search.sh:/etc/temporal/entrypoint.sh

  temporal-ui:
    container_name: temporal-ui
    restart: unless-stopped
    depends_on:
      - temporal
    environment:
      - TEMPORAL_ADDRESS=temporal:7233
      - TEMPORAL_CORS_ORIGINS=http://localhost:3000
      - TEMPORAL_CSRF_COOKIE_INSECURE=true
    image: temporalio/ui:2.35.1@sha256:a98e49436dab613aaea77e72da4003f606045deaca066b706c421eeb2c1acad4
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - 8085:8080

The full logs when Docker compose starts:

temporal              | TEMPORAL_ADDRESS is not set, setting it to 192.168.107.3:7233
temporal              | PostgreSQL started.
temporal              | Setup PostgreSQL schema.
temporal-ui           | 2025/03/03 10:27:55 Loading config; env=docker,configDir=config
temporal-ui           | 2025/03/03 10:27:55 Loading config files=[config/docker.yaml]
temporal-ui           | 2025/03/03 10:27:55 Loading config; env=docker,configDir=config
temporal-ui           | 2025/03/03 10:27:55 Loading config files=[config/docker.yaml]
temporal-ui           |
temporal-ui           |    ____    __
temporal-ui           |   / __/___/ /  ___
temporal-ui           |  / _// __/ _ \/ _ \
temporal-ui           | /___/\__/_//_/\___/ v4.9.0
temporal-ui           | High performance, minimalist Go web framework
temporal-ui           | https://echo.labstack.com
temporal-ui           | ____________________________________O/_______
temporal-ui           |                                     O\
temporal-ui           | ⇨ http server started on [::]:8080
peerdb-server         | 2025-03-03T10:27:56.481070Z  INFO NOTICE: relation "refinery_schema_history" already exists, skipping
peerdb-server         | 2025-03-03T10:27:56.777997Z  INFO current version: 42
peerdb-server         | 2025-03-03T10:27:56.778073Z  INFO no migrations to apply
peerdb-server         | 2025-03-03T10:27:56.778908Z  INFO Listening on 0.0.0.0:9900
peerdb-server         | 2025-03-03T10:27:56.778924Z  INFO MIRROR commands enabled
peerdb-server         | 2025-03-03T10:27:56.778930Z  INFO connecting to flow server at grpc://flow_api:8112/grpc
temporal              | 2025-03-03T10:27:57.065Z	INFO	Starting schema setup	{"config": {"SchemaFilePath":"","SchemaName":"","InitialVersion":"0.0","Overwrite":false,"DisableVersioning":false}, "logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/setuptask.go:64"}
temporal              | 2025-03-03T10:27:57.066Z	DEBUG	Setting up version tables	{"logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/setuptask.go:74"}
temporal              | 2025-03-03T10:27:57.262Z	DEBUG	Current database schema version 1.15 is greater than initial schema version 0.0. Skip version upgrade	{"logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/setuptask.go:135"}
temporal              | 2025-03-03T10:27:57.262Z	INFO	Schema setup complete	{"logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/setuptask.go:150"}
temporal              | 2025-03-03T10:27:57.784Z	INFO	UpdateSchemaTask started	{"config": {"DBName":"","TargetVersion":"","SchemaDir":"/etc/temporal/schema/postgresql/v12/temporal/versioned","SchemaName":"","IsDryRun":false}, "logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/updatetask.go:106"}
temporal              | 2025-03-03T10:27:57.885Z	DEBUG	Schema Dirs: []	{"logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/updatetask.go:214"}
temporal              | 2025-03-03T10:27:57.885Z	DEBUG	found zero updates from current version 1.15	{"logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/updatetask.go:136"}
temporal              | 2025-03-03T10:27:57.886Z	INFO	UpdateSchemaTask done	{"logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/updatetask.go:129"}
temporal              | 2025-03-03T10:27:58.971Z	INFO	Starting schema setup	{"config": {"SchemaFilePath":"","SchemaName":"","InitialVersion":"0.0","Overwrite":false,"DisableVersioning":false}, "logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/setuptask.go:64"}
temporal              | 2025-03-03T10:27:58.972Z	DEBUG	Setting up version tables	{"logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/setuptask.go:74"}
temporal              | 2025-03-03T10:27:59.170Z	DEBUG	Current database schema version 1.9 is greater than initial schema version 0.0. Skip version upgrade	{"logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/setuptask.go:135"}
temporal              | 2025-03-03T10:27:59.170Z	INFO	Schema setup complete	{"logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/setuptask.go:150"}
temporal              | 2025-03-03T10:27:59.675Z	INFO	UpdateSchemaTask started	{"config": {"DBName":"","TargetVersion":"","SchemaDir":"/etc/temporal/schema/postgresql/v12/visibility/versioned","SchemaName":"","IsDryRun":false}, "logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/updatetask.go:106"}
temporal              | 2025-03-03T10:27:59.774Z	DEBUG	Schema Dirs: []	{"logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/updatetask.go:214"}
temporal              | 2025-03-03T10:27:59.774Z	DEBUG	found zero updates from current version 1.9	{"logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/updatetask.go:136"}
temporal              | 2025-03-03T10:27:59.774Z	INFO	UpdateSchemaTask done	{"logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/tools/common/schema/updatetask.go:129"}
temporal              | Temporal CLI address: 192.168.107.3:7233.
temporal              | 2025/03/03 10:27:59 Loading config; env=docker,zone=,configDir=config
temporal              | 2025/03/03 10:27:59 Loading config files=[config/docker.yaml]
temporal              | {"level":"info","ts":"2025-03-03T10:27:59.836Z","msg":"Build info.","git-time":"2025-02-21T21:43:57.000Z","git-revision":"675b2d5a8a2c9c6051f618e3728c6082fa925078","git-modified":false,"go-arch":"arm64","go-os":"linux","go-version":"go1.23.2","cgo-enabled":false,"server-version":"1.27.0","debug-mode":false,"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/cmd/server/main.go:175"}
temporal              | {"level":"info","ts":"2025-03-03T10:27:59.837Z","msg":"dynamic config changed for the key: limit.maxidlength oldValue: nil newValue: { constraints: {} value: 255 }","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:361"}
temporal              | {"level":"info","ts":"2025-03-03T10:27:59.837Z","msg":"dynamic config changed for the key: system.forcesearchattributescacherefreshonread oldValue: nil newValue: { constraints: {} value: true }","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:361"}
temporal              | {"level":"info","ts":"2025-03-03T10:27:59.837Z","msg":"dynamic config changed for the key: frontend.enableupdateworkflowexecution oldValue: nil newValue: { constraints: {} value: true }","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:361"}
temporal              | {"level":"info","ts":"2025-03-03T10:27:59.837Z","msg":"Updated dynamic config","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:207"}
temporal              | {"level":"warn","ts":"2025-03-03T10:27:59.837Z","msg":"Not using any authorizer and flag `--allow-no-auth` not detected. Future versions will require using the flag `--allow-no-auth` if you do not want to set an authorizer.","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/cmd/server/main.go:205"}
temporal              | time=2025-03-03T10:27:59.838 level=ERROR msg="failed reaching server: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 192.168.107.3:7233: connect: connection refused\""
temporal              | Waiting for Temporal server to start...
temporal              | [Fx] PROVIDE	fx.Lifecycle <= go.uber.org/fx.New.func1()
temporal              | [Fx] PROVIDE	fx.Shutdowner <= go.uber.org/fx.(*App).shutdowner-fm()
temporal              | [Fx] PROVIDE	fx.DotGraph <= go.uber.org/fx.(*App).dotGraph-fm()
temporal              | [Fx] PROVIDE	*temporal.ServerImpl <= go.temporal.io/server/temporal.NewServerFxImpl()
temporal              | [Fx] PROVIDE	*temporal.serverOptions <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	chan interface {} <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	temporal.synchronizationModeParams <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	*config.Config <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	*config.PProf <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	log.Config <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	resource.ServiceNames <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	resource.NamespaceLogger <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	resolver.ServiceResolver <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	client.AbstractDataStoreFactory <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	visibility.VisibilityStoreFactory <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	searchattribute.Mapper <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	[]grpc.UnaryServerInterceptor <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	authorization.Authorizer <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	authorization.ClaimMapper <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	authorization.JWTAudienceMapper <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	map[primitives.ServiceName]static.Hosts <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	log.Logger <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	client.FactoryProvider <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	dynamicconfig.Client <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	encryption.TLSConfigProvider <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	*client.Config <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	client.Client <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	metrics.Handler <= go.temporal.io/server/temporal.ServerOptionsProvider()
temporal              | [Fx] PROVIDE	archiver.ArchivalMetadata <= go.temporal.io/server/common/resource.ArchivalMetadataProvider()
temporal              | [Fx] PROVIDE	tasks.TaskCategoryRegistry <= go.temporal.io/server/temporal.TaskCategoryRegistryProvider()
temporal              | [Fx] PROVIDE	client.FactoryProviderFn <= go.temporal.io/server/temporal.PersistenceFactoryProvider()
temporal              | [Fx] PROVIDE	*temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.HistoryServiceProvider()
temporal              | [Fx] PROVIDE	*temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.MatchingServiceProvider()
temporal              | [Fx] PROVIDE	*temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.FrontendServiceProvider()
temporal              | [Fx] PROVIDE	*temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.InternalFrontendServiceProvider()
temporal              | [Fx] PROVIDE	*temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.WorkerServiceProvider()
temporal              | [Fx] PROVIDE	*cluster.Config <= go.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider()
temporal              | [Fx] PROVIDE	config.Persistence <= go.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider()
temporal              | [Fx] PROVIDE	*dynamicconfig.Collection <= go.temporal.io/server/common/dynamicconfig.init.func1()
temporal              | [Fx] PROVIDE	pingable.Pingable[group = "deadlockDetectorRoots"] <= fx.Annotate(go.temporal.io/server/common/dynamicconfig.init.func2(), fx.ResultTags(["group:\"deadlockDetectorRoots\""])
temporal              | [Fx] PROVIDE	*pprof.PProfInitializerImpl <= go.temporal.io/server/common/pprof.NewInitializer()
temporal              | [Fx] PROVIDE	[]trace.SpanExporter <= go.temporal.io/server/temporal.init.func2()
temporal              | [Fx] SUPPLY	[]temporal.ServerOption
temporal              | [Fx] RUN	supply: stub([]temporal.ServerOption) in 4.75µs
temporal              | [Fx] RUN	provide: go.temporal.io/server/temporal.ServerOptionsProvider() in 101.082576ms
temporal              | [Fx] Error returned: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider
temporal              | 	/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:182:
temporal              | sql schema version compatibility check failed: unable to read DB schema version keyspace/database: temporal error: no usable database connection found
temporal              | [Fx] ERROR		Failed to initialize custom logger: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2
temporal              | 	/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/module.go:294:
temporal              | failed to build fxevent.Logger:
temporal              | could not build arguments for function "go.temporal.io/server/temporal".init.func7
temporal              | 	/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1008:
temporal              | failed to build log.Logger:
temporal              | received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider
temporal              | 	/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:182:
temporal              | sql schema version compatibility check failed: unable to read DB schema version keyspace/database: temporal error: no usable database connection found
temporal              | Unable to create server. Error: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2 (/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/module.go:294): failed to build fxevent.Logger: could not build arguments for function "go.temporal.io/server/temporal".init.func7 (/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1008): failed to build log.Logger: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider (/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:182): sql schema version compatibility check failed: unable to read DB schema version keyspace/database: temporal error: no usable database connection found.
temporal exited with code 1
temporal              | PostgreSQL started.
temporal              | Setup PostgreSQL schema.
temporal-admin-tools  | time=2025-03-03T10:28:01.036 level=ERROR msg="failed reaching server: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 192.168.107.3:7233: connect: connection refused\""
temporal-admin-tools  | time=2025-03-03T10:28:01.050 level=ERROR msg="failed reaching server: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 192.168.107.3:7233: connect: connection refused\""

Thank you for any help! I’m really stuck there. Thank you! :heart:

I managed to make it work. The auto-setup container uses 2 different sets of ENV variables for configuring TLS for Postgres connection:

POSTGRES_TLS_ENABLED, POSTGRES_TLS_DISABLE_HOST_VERIFICATION & POSTGRES_TLS_CA_FILE ENV variables are used to setup the databases - create DB, check version, execute migrations, …

SQL_TLS_ENABLED, SQL_HOST_VERIFICATION & SQL_CA ENV variables are use then later on for actual run of the Temporal server within the container.

That’s why the DB setup got excuted correctly, but then the server crashed. I didn’t configure the second set of ENV variables so it was using incorrect TLS configuration for Postgres connection.

Maybe it’s worth to unify it so it’s not that confusing? :folded_hands: