Kubernetes + Helm + Nginx Ingress in combination with Temporal TLS not working. Help needed

Community Support
, , ,
1

Hi guys,

we are trying to run temporal on Kubernetes with the provided helm chart example of temporal. But we are struggling a lot with the proper configuration when it comes to TLS. What we got so far is that both nginx ingresses are running (the grpc frontend and the web ui). TLS termination works fine and no problems at all. Until the point we follow the guideline to enable also TLS for the frontend service and internode communication. We found out that as soon as we configured tls, the web ui receives an internal server error and you cant find any helpful logs in the pods. As soon as we figured out that you need to enable TLS passthrough for the frontend ingress, at least the UI was then working again and workers could also connect to the grpc frontend. But unfortunately the worker deployment itself in the temporal cluster wasnt able to connect anymore. So it went into a crash loopback. In the Web UI, I receive an 503 for one REST call to the namespaces endpoint. So in general speaking, as soon as we enable TLS for the frontend, it doesnt seem to harmonize with the ingress or the internal communication.

Here are some configs we use:

Web Ingress

web:
  ingress:
    enabled: true
    className: nginx
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
      cert-manager.io/issuer: "temporal-issuer"
    hosts:
      - "temporal.ui.mydomain.com"
    tls:
      - secretName: alf-temporal-tls
        hosts:
          - temporal.ui.mydomain.com

Frontend Ingress

server:
  frontend:
    ingress:
      enabled: true
      className: nginx
      annotations: 
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/ssl-redirect: "true"
        nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
        nginx.ingress.kubernetes.io/ssl-passthrough: "true"
        nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
        cert-manager.io/issuer: "temporal-issuer"
      hosts:
        - "temporal.grpc.mydomain.com"
      tls: 
        - secretName: alf-temporal-grpc
          hosts:
            - temporal.grpc.mydomain.com

TLS config in Helm chart

 tls:
      frontend:
        server:
          certFile: /etc/cert/tls.crt
          keyFile: /etc/cert/tls.key
        client:
          serverName: temporal.grpc.mydomain.com

Note that if I dont provide in the additional env of the web the server name, the UI returns a 500 status code without any logs. It seems also that it doesnt matter what name I put in there. It just has to be set?!

web:
  additionalEnv:
    - name: TEMPORAL_TLS_SERVER_NAME
      value: <any url seems to work here>

A couple questions would be interesting, that could help to solve:

  • Do we need to configure the services differently in the kubernetes cluster?
  • What is the proper hostname for the frontend and internode TLS ?
  • Where can I set the temporal hostname or where can I see it? All names I could find are internal cluster names and not a real domain
  • What CN / DNS alias should the certificates contain for frontend and internode?
  • Why do I need to provide the TLS servername in the web as additional env? This is not documented anywhere.
2

You may also need to insert additional env parameters like

web:
  image:
    tag: 2.29.2
  additionalEnv:
    - name: TEMPORAL_TLS_CA
      value: /etc/certs/temporal/ca.cert
    - name: TEMPORAL_TLS_CERT
      value: /etc/certs/temporal/cluster.pem
    - name: TEMPORAL_TLS_KEY
      value: /etc/certs/temporal/cluster.key
    - name: TEMPORAL_TLS_ENABLE_HOST_VERIFICATION
      value: "false"