Error after upgrading from 1.14.4 to 1.15.2

Community Support
1

I am testing an upgrade from v1.14.4 to v1.15.2 and run into an issue when TLS is enabled. The temporal server will not start with the following exception.

{"level":"fatal","ts":"2022-03-16T15:20:17.671Z","msg":"error starting scanner","service":"worker","error":"context deadline exceeded","logging-call-at":"service.go:436","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Fatal\n\t/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/service/worker.(*Service).startScanner\n\t/temporal/service/worker/service.go:436\ngo.temporal.io/server/service/worker.(*Service).Start\n\t/temporal/service/worker/service.go:343\ngo.temporal.io/server/service/worker.ServiceLifetimeHooks.func1.1\n\t/temporal/service/worker/fx.go:79"}

We did not change the TLS configuration in config when upgrading. Was there a change to the TLS configuration that we missed? I saw a post that recommended adding systemWorker configuration but that did not seem to fix the issue either. Here is our TLS config that we used successfully on 1.14.4

    tls:
        refreshInterval: {{ default .Env.TEMPORAL_TLS_REFRESH_INTERVAL "0s" }}
        expirationChecks:
            warningWindow: {{ default .Env.TEMPORAL_TLS_EXPIRATION_CHECKS_WARNING_WINDOW "0s" }}
            errorWindow: {{ default .Env.TEMPORAL_TLS_EXPIRATION_CHECKS_ERROR_WINDOW "0s" }}
            checkInterval: {{ default .Env.TEMPORAL_TLS_EXPIRATION_CHECKS_CHECK_INTERVAL "0s" }}
        internode:
            # This server section configures the TLS certificate that internal temporal
            # cluster nodes (history or matching) present to other clients within the Temporal Cluster.
            server:
                requireClientAuth: {{ default .Env.TEMPORAL_TLS_REQUIRE_CLIENT_AUTH "false" }}

                certFile: {{ default .Env.TEMPORAL_TLS_SERVER_CERT "" }}
                keyFile: {{ default .Env.TEMPORAL_TLS_SERVER_KEY "" }}
                {{- if .Env.TEMPORAL_TLS_SERVER_CA_CERT }}
                clientCaFiles:
                    - {{ default .Env.TEMPORAL_TLS_SERVER_CA_CERT "" }}
                {{- end }}

                certData: {{ default .Env.TEMPORAL_TLS_SERVER_CERT_DATA "" }}
                keyData: {{ default .Env.TEMPORAL_TLS_SERVER_KEY_DATA "" }}
                {{- if .Env.TEMPORAL_TLS_SERVER_CA_CERT_DATA }}
                clientCaData:
                    - {{ default .Env.TEMPORAL_TLS_SERVER_CA_CERT_DATA "" }}
                {{- end }}

            # This client section is used to configure the TLS clients within
            # the Temporal Cluster that connect to an Internode (history or matching)
            client:
                serverName: "{{ default .Env.TEMPORAL_TLS_INTERNODE_SERVER_NAME "" }}"
                disableHostVerification: {{ default .Env.TEMPORAL_TLS_INTERNODE_DISABLE_HOST_VERIFICATION "false"}}
                {{- if .Env.TEMPORAL_TLS_SERVER_CA_CERT }}
                rootCaFiles:
                    - {{ default .Env.TEMPORAL_TLS_SERVER_CA_CERT "" }}
                {{- end }}
                {{- if .Env.TEMPORAL_TLS_SERVER_CA_CERT_DATA }}
                rootCaData:
                    - {{ default .Env.TEMPORAL_TLS_SERVER_CA_CERT_DATA "" }}
                {{- end }}
        frontend:
            # This server section configures the TLS certificate that the Frontend
            # server presents to all clients (specifically the Worker role within
            # the Temporal Cluster and all External SDKs connecting to the Cluster)
            server:
                requireClientAuth: {{ default .Env.TEMPORAL_TLS_REQUIRE_CLIENT_AUTH "false" }}
                certFile: {{ default .Env.TEMPORAL_TLS_FRONTEND_CERT "" }}
                keyFile: {{ default .Env.TEMPORAL_TLS_FRONTEND_KEY "" }}
                {{- if .Env.TEMPORAL_TLS_CLIENT1_CA_CERT }}
                clientCaFiles:
                    - {{ default .Env.TEMPORAL_TLS_CLIENT1_CA_CERT "" }}
                    - {{ default .Env.TEMPORAL_TLS_CLIENT2_CA_CERT "" }}
                {{- end }}

                certData: {{ default .Env.TEMPORAL_TLS_FRONTEND_CERT_DATA "" }}
                keyData: {{ default .Env.TEMPORAL_TLS_FRONTEND_KEY_DATA "" }}
                {{- if .Env.TEMPORAL_TLS_CLIENT1_CA_CERT_DATA }}
                clientCaData:
                    - {{ default .Env.TEMPORAL_TLS_CLIENT1_CA_CERT_DATA "" }}
                    - {{ default .Env.TEMPORAL_TLS_CLIENT2_CA_CERT_DATA "" }}
                {{- end }}

            # This client section is used to configure the TLS clients within
            # the Temporal Cluster (specifically the Worker role) that connect to the Frontend service
            client:
                serverName: "{{ default .Env.TEMPORAL_TLS_FRONTEND_SERVER_NAME "" }}"
                disableHostVerification: {{ default .Env.TEMPORAL_TLS_FRONTEND_DISABLE_HOST_VERIFICATION "false"}}
                {{- if .Env.TEMPORAL_TLS_SERVER_CA_CERT }}
                rootCaFiles:
                    - {{ default .Env.TEMPORAL_TLS_SERVER_CA_CERT "" }}
                {{- end }}
                {{- if .Env.TEMPORAL_TLS_SERVER_CA_CERT_DATA }}
                rootCaData:
                    - {{ default .Env.TEMPORAL_TLS_SERVER_CA_CERT_DATA "" }}
                {{- end }}

Thanks

2

Looks like it was this forceTLS config value How to configure TLS for the internal System Worker · Issue #933 · temporalio/documentation · GitHub from this commit : Treat enablement of TLS separately for server and client config (#2501) · temporalio/temporal@632c4f7 · GitHub

1 Like