Serverjwtauth temporal start failed

Community Support Server Deployment
, ,
1

I am following Temporal JWT Authorization this example and trying to make the JWT work.

I started the jwks server and change the port to 61884.

go run ./serverjwtauth/key gen-and-serve

I used the docker-compose.yml and added 4 env to temporal and 1 env to temporal-admin-tools

  temporal:
    container_name: temporal
    depends_on:
      - postgresql
      - elasticsearch
    environment:
...
      - TEMPORAL_JWT_KEY_SOURCE1=http://host.docker.internal:61884/jwks.json
      - TEMPORAL_AUTH_AUTHORIZER=default
      - TEMPORAL_AUTH_CLAIM_MAPPER=default
      - TEMPORAL_CLI_AUTH=Bearer eyJhbGciOiJFUzI1NiIsImtpZCI6IkdXRUtVemFJY0EzOGx0RlpYM1lfNGpJaHFwbU4xRG9GeW9tMUk4WFBmU3M9IiwidHlwIjoiSldUIn0.eyJleHAiOjE2OTYxNDkzNjUsInBlcm1pc3Npb25zIjpbInN5c3RlbTphZG1pbiJdLCJzdWIiOiJ0ZW1wb3JhbC1zYW1wbGVzLWdvIn0.PCIETv1EHH2y2H7bpQa2McY-0uhsUmPT6BWRli7k0knmWqCECF3q_USMxJOGr16SlB3Q7yRCDKSNPSbG7wqevQ
    image: temporalio/auto-setup:${TEMPORAL_VERSION}
...


  temporal-admin-tools:
    container_name: temporal-admin-tools
    depends_on:
      - temporal
    environment:
      - TEMPORAL_ADDRESS=temporal:7233 # used by temporal
      - TEMPORAL_CLI_ADDRESS=temporal:7233 # used by tctl
      - TEMPORAL_CLI_AUTH=Bearer eyJhbGciOiJFUzI1NiIsImtpZCI6IkdXRUtVemFJY0EzOGx0RlpYM1lfNGpJaHFwbU4xRG9GeW9tMUk4WFBmU3M9IiwidHlwIjoiSldUIn0.eyJleHAiOjE2OTYxNDkzNjUsInBlcm1pc3Npb25zIjpbInN5c3RlbTphZG1pbiJdLCJzdWIiOiJ0ZW1wb3JhbC1zYW1wbGVzLWdvIn0.PCIETv1EHH2y2H7bpQa2McY-0uhsUmPT6BWRli7k0knmWqCECF3q_USMxJOGr16SlB3Q7yRCDKSNPSbG7wqevQ
    image: temporalio/admin-tools:${TEMPORAL_VERSION}
...

After running the compose file, the temporal container can’t start, and I see this error.

{"level":"fatal","ts":"2023-10-01T08:00:18.820Z","msg":"error starting scanner","service":"worker","error":"Request unauthorized.","logging-call-at":"service.go:501","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Fatal\n\t/home/builder/temporal/common/log/zap_logger.go:174\ngo.temporal.io/server/service/worker.(*Service).startScanner\n\t/home/builder/temporal/service/worker/service.go:501\ngo.temporal.io/server/service/worker.(*Service).Start\n\t/home/builder/temporal/service/worker/service.go:388\ngo.temporal.io/server/service/worker.ServiceLifetimeHooks.func1.1\n\t/home/builder/temporal/service/worker/fx.go:139"}

I tried to create the namespace with tctl, but since the temporal server start failed, I couldn’t create inside the temporal-admin-tools terminal, it said connection error.

Error: Register namespace operation failed.
Error Details: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)
2

the README.md said need to set cli auth for tctl and register a default namespace. Doesn’t tctl need to talk to a temporal server? But how to do that when the temporal server is not up successfully in the first place?

3

please check this PR if you have the same problem =)

4

Hi @luis_chan , I came up with the same issue.
After applying your PR changes, I was able to make temporal up and running finally.
Yet, still every request I sent gives unauthorized error :

$> tctl --ns default namespace register -rd 1
Error: Register namespace operation failed.
Error Details: rpc error: code = PermissionDenied desc = Request unauthorized.
('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)

That might be ok, since I had already created default namespace before Authorization enabling. but this one also causes error :

$ go run ./serverjwtauth/worker
on server side : 
temporal                | {"level":"error","ts":"2024-01-04T22:39:04.021Z","msg":"Authorization error","error":"ECDSA key not found for key ID: 18SFGmWkNtbmCfUOvNxy6IBDnTrdQrG_PqoHL_xCV14=","logging-call-at":"interceptor.go:174","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/authorization.(*interceptor).logAuthError\n\t/home/builder/temporal/common/authorization/interceptor.go:174\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/home/builder/temporal/common/authorization/interceptor.go:120\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/builder/temporal/common/rpc/interceptor/telemetry.go:165\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).handleLocalAPIInvocation\n\t/home/builder/temporal/service/frontend/redirection_interceptor.go:214\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).Intercept\n\t/home/builder/temporal/service/frontend/redirection_interceptor.go:188\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/home/builder/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.42.0/interceptor.go:344\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_logger.go:84\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:111\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/home/builder/temporal/common/rpc/grpc.go:145\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1186\ngo.temporal.io/api/workflowservice/v1._WorkflowService_GetSystemInfo_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.24.1-0.20231003165936-bb03061759c8/workflowservice/v1/service.pb.go:2239\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1376\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1753\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:998"}
5

how to set TEMPORAL_CLI_AUTH =Bearer xxxxx in cli (GitHub - temporalio/cli: Command-line interface for running Temporal Server and interacting with Workflows, Activities, Namespaces, and other parts of Temporal)?
or does cli don’t support TEMPORAL_CLI_AUTH and we need to use tctl only?