Struggling to get ingress working for frontend gRPC service

I’ve deployed Temporal on K8s and I’m not able to connect to the frontend gRPC service from outside the cluster. The ingress I’ve set up works for the web service, but I can’t seem to get it to work for gRPC. I’m using the https://github.com/kubernetes/ingress-nginx ingress controller with a network load balancer on an AWS EKS cluster.

When I run:

kubectl port-forward temporaltest-frontend-fb6b79468-cznvc 7233:7233

and then in another window:

❯ grpcurl -plaintext localhost:7233 list 
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
temporal.api.workflowservice.v1.WorkflowService
temporal.server.api.adminservice.v1.AdminService

I’m able to see the services on port 7233. This would seem to indicate that the service itself is running without issue and that the issue is with the Ingress resource or the ingress controller resource.

Here’s what I see if I try to grpcurl my endpoint:

❯ grpcurl -plaintext ssokolin-test.useavenue.com:443 list
Failed to dial target host "ssokolin-test.useavenue.com:443": context deadline exceeded

Any help here would be very much appreciated!!

My Ingress resource manifest is below:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    ingressclass.kubernetes.io/is-default-class: 'true'
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'false'
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '60'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '60'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '60'
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    nginx.org/grpc-services: temporaltest-frontend
  labels:
    app.kubernetes.io/component: web
    app.kubernetes.io/instance: temporaltest
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: temporal
    app.kubernetes.io/part-of: temporal
    app.kubernetes.io/version: 1.14.0
    helm.sh/chart: temporal-0.14.0
  name: temporaltest-web
spec:
  rules:
    - host: ssokolin-test.useavenue.com
      http:
        paths:
          - backend:
              serviceName: temporaltest-frontend
              servicePort: 7233
            path: /
  tls:
    - hosts:
        - ssokolin-test.useavenue.com
      secretName: secret1

My Temporal frontend Service manifest is below:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: 'true'
  labels:
    app.kubernetes.io/component: frontend
    app.kubernetes.io/headless: 'true'
    app.kubernetes.io/instance: temporaltest
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: temporal
    app.kubernetes.io/part-of: temporal
    app.kubernetes.io/version: 1.14.0
    helm.sh/chart: temporal-0.14.0
    prometheus.io/job: temporal-frontend
    prometheus.io/port: '9090'
    prometheus.io/scheme: http
    prometheus.io/scrape: 'true'
  name: temporaltest-frontend-headless
spec:
  clusterIP: None
  ports:
    - name: grpc-rpc
      port: 7233
      protocol: TCP
      targetPort: rpc
    - name: metrics
      port: 9090
      protocol: TCP
      targetPort: metrics
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/component: frontend
    app.kubernetes.io/instance: temporaltest
    app.kubernetes.io/name: temporal
  type: ClusterIP

My Ingress controller Service manifest is below:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.43.0
    helm.sh/chart: ingress-nginx-3.20.1
  name: nginx-ingress-ingress-nginx-controller
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer

do you get the same message with authority header in this grpcurl command:
-authority

what about if you use ip directly?

Yeah I get the same message, both for authority and using ip directly

It seems like ingress nginx is expecting tls (evidence by port 443 and the tls secret on the K8s Ingress Object). Can you try grpcurl specifying the client cert and key?

Ah I realize you’re not trying to do mtls auth. Are there any logs in either nginx or front end pod when attempting to connect you could share? Can you try port forwarding to the service instead of directly to pod?

Port forwarding to the service works:

❯ kubectl port-forward service/temporaltest-frontend 7233:7233
Forwarding from 127.0.0.1:7233 -> 7233
Forwarding from [::1]:7233 -> 7233
Handling connection for 7233
❯ grpcurl -plaintext localhost:7233 list
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
temporal.api.workflowservice.v1.WorkflowService
temporal.server.api.adminservice.v1.AdminService

I got it!!

❯ grpcurl temporal-frontend.ssokolin-test.com:443 list
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
temporal.api.workflowservice.v1.WorkflowService
temporal.server.api.adminservice.v1.AdminService

I’m pretty sure the issue was that I was getting rate-limited by the TLS Certificate Authority

When I switched to their staging url, it worked:

1 Like

Thanks for the update! Best wishes!