I am trying to setup a temporal server on our environment. After the server is started, I see the container going for a bootloop with following error:
Error: unable to health check "temporal.api.workflowservice.v1.WorkflowService" service: connection error: desc = "transport: Error while dialing: dial tcp 10.255.0.6:7233: connect: connection refused"
Later, I get this error, after which the service continuously gets restarted.
{ [-]
ec2: { [+]
}
env: ddev
error: failed reaching server: last connection error: connection error: desc = "error reading server preface: http2: frame too large"
level: fatal
logging-call-at: factory.go:121
m: { [+]
}
micros_container: temporal
msg: error creating sdk client
service: worker
stacktrace: go.temporal.io/server/common/log.(*zapLogger).Fatal
/home/builder/temporal/common/log/zap_logger.go:180
go.temporal.io/server/common/sdk.(*clientFactory).GetSystemClient.func1
/home/builder/temporal/common/sdk/factory.go:121
sync.(*Once).doSlow
/usr/local/go/src/sync/once.go:74
sync.(*Once).Do
/usr/local/go/src/sync/once.go:65
go.temporal.io/server/common/sdk.(*clientFactory).GetSystemClient
/home/builder/temporal/common/sdk/factory.go:108
go.temporal.io/server/service/worker/scanner.(*Scanner).Start
/home/builder/temporal/service/worker/scanner/scanner.go:229
go.temporal.io/server/service/worker.(*Service).startScanner
/home/builder/temporal/service/worker/service.go:500
go.temporal.io/server/service/worker.(*Service).Start
/home/builder/temporal/service/worker/service.go:396
go.uber.org/fx/internal/lifecycle.Wrap[...].func1
/go/pkg/mod/go.uber.org/fx@v1.20.0/internal/lifecycle/lifecycle.go:80
go.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook
/go/pkg/mod/go.uber.org/fx@v1.20.0/internal/lifecycle/lifecycle.go:256
go.uber.org/fx/internal/lifecycle.(*Lifecycle).Start
/go/pkg/mod/go.uber.org/fx@v1.20.0/internal/lifecycle/lifecycle.go:216
go.uber.org/fx.(*App).start.func1
/go/pkg/mod/go.uber.org/fx@v1.20.0/app.go:679
go.uber.org/fx.(*App).withRollback
/go/pkg/mod/go.uber.org/fx@v1.20.0/app.go:661
go.uber.org/fx.(*App).start
/go/pkg/mod/go.uber.org/fx@v1.20.0/app.go:678
go.uber.org/fx.withTimeout.func1
/go/pkg/mod/go.uber.org/fx@v1.20.0/app.go:782
time: 2024-04-08T07:11:44.323879113Z
ts: 2024-04-08T07:11:44.323Z
}
I tried to look at the docker.yaml at /etc/temporal/config, it seem to be looking good
log:
stdout: true
level: debug
persistence:
numHistoryShards: 4
defaultStore: default
visibilityStore: visibility
datastores:
default:
sql:
pluginName: "postgres"
databaseName: "XXXX"
connectAddr: "XXXX"
connectProtocol: "tcp"
user: "XXXX"
password: "XXXX"
maxConns: 20
maxIdleConns: 20
maxConnLifetime: 1h
tls:
enabled: true
caFile:
certFile:
keyFile:
enableHostVerification: false
serverName:
visibility:
sql:
pluginName: "postgres"
databaseName: "XXXX"
connectAddr: "XXXX"
connectProtocol: "tcp"
user: "XXXX"
password: "XXXX"
maxConns: 10
maxIdleConns: 10
maxConnLifetime: 1h
tls:
enabled: true
caFile:
certFile:
keyFile:
enableHostVerification: false
serverName:
global:
membership:
maxJoinDuration: 30s
broadcastAddress: "172.29.191.22"
pprof:
port: 0
tls:
refreshInterval: 0s
expirationChecks:
warningWindow: 0s
errorWindow: 0s
checkInterval: 0s
internode:
# This server section configures the TLS certificate that internal temporal
# cluster nodes (history, matching, and internal-frontend) present to other
# clients within the Temporal Cluster.
server:
requireClientAuth: false
certFile:
keyFile:
certData:
keyData:
# This client section is used to configure the TLS clients within
# the Temporal Cluster that connect to an Internode (history, matching, or
# internal-frontend)
client:
serverName:
disableHostVerification: false
frontend:
# This server section configures the TLS certificate that the Frontend
# server presents to external clients.
server:
requireClientAuth: false
certFile:
keyFile:
certData:
keyData:
# This client section is used to configure the TLS clients within
# the Temporal Cluster (specifically the Worker role) that connect to the Frontend service
client:
serverName:
disableHostVerification: false
authorization:
jwtKeyProvider:
keySourceURIs:
refreshInterval: 1m
permissionsClaimName: permissions
authorizer:
claimMapper:
services:
frontend:
rpc:
grpcPort: 8080
membershipPort: 6933
bindOnIP: 0.0.0.0
httpPort: 7243
matching:
rpc:
grpcPort: 7235
membershipPort: 6935
bindOnIP: 0.0.0.0
history:
rpc:
grpcPort: 7234
membershipPort: 6934
bindOnIP: 0.0.0.0
worker:
rpc:
grpcPort: 7239
membershipPort: 6939
bindOnIP: 0.0.0.0
clusterMetadata:
enableGlobalNamespace: false
failoverVersionIncrement: 10
masterClusterName: "active"
currentClusterName: "active"
clusterInformation:
active:
enabled: true
initialFailoverVersion: 1
rpcName: "frontend"
rpcAddress: 127.0.0.1:8080
dcRedirectionPolicy:
policy: "noop"
archival:
history:
state: "enabled"
enableRead: true
provider:
filestore:
fileMode: "0666"
dirMode: "0766"
visibility:
state: "enabled"
enableRead: true
provider:
filestore:
fileMode: "0666"
dirMode: "0766"
namespaceDefaults:
archival:
history:
state: "disabled"
URI: "file:///tmp/temporal_archival/development"
visibility:
state: "disabled"
URI: "file:///tmp/temporal_vis_archival/development"
dynamicConfigClient:
filepath: "/etc/temporal/config/dynamicconfig/docker.yaml"
pollInterval: "60s"
Is it something to do with the tls certificates and verification?
Additional details:
I am using temporalio/auto-setup:1.22.0 and using postgres for the data persistence