Temporal cli cannot connect to frontend service via Ingress

Community Support Server Deployment
,
1

I deployed Temporal Cluster on on-prem K8s. My deployment is based on Helm Chart with some modifications.
I created Ingress for exposing frontend service outside of K8s, but Temporal cli tool fails to connect. All works like a charm if I expose service with port forwarding.

Ingress config:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'false'
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '60'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '60'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '60'
    nginx.ingress.kubernetes.io/ssl-passthrough: 'false'
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    nginx.org/grpc-services: temporal-frontend
  name: temporal-frontend
  namespace: temporal-stage
spec:
  rules:
    - host: api.temporal.stage.stellar-dev.detmir-infra.ru
      http:
        paths:
          - backend:
              service:
                name: temporal-frontend
                port:
                  name: grpc-rpc
            path: /
            pathType: Prefix
  tls:
    - hosts:
        - api.temporal.stage.stellar-dev.detmir-infra.ru
      secretName: temporal-frontend-cert

I was able to connect to external address using grpcurl:

 grpcurl api.temporal.stage.stellar-dev.detmir-infra.ru:443 list

grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
temporal.api.operatorservice.v1.OperatorService
temporal.api.workflowservice.v1.WorkflowService
temporal.server.api.adminservice.v1.AdminService

But when I triyng to connect with temporal cli I got an error:

 temporal workflow list --env stage

 FATAL   Failed to create SDK client     {"error": "failed reaching server: last connection error: connection error: desc = \"error reading server preface: http2: frame too large\"", "logging-call-at": "factory.go:141"}

 temporal env get stage

address  api.temporal.stage.stellar-dev.detmir-infra.ru:443
2

Were you able to get to the bottom of this issue? Can you tell from service_requests server metric if operation ListWorkflowExecutions was received by frontend? If not maybe was blocked with this error by nginx?

3

+1, same problem

4

Yes, sorry for long delay.
The issue was with LetsEncrypt certificate which is not recognized as valid certificate authority by CLI tool.
Solved by adding parameter --tls-ca-path referencing the .pem file with root LetsEncrypt certificate. It turns out that this certificate is also required to connect with Java SDK.

1 Like