Struggling to get ingress working for frontend gRPC service

Community Support
,
1

I’ve deployed Temporal on K8s and I’m not able to connect to the frontend gRPC service from outside the cluster. The ingress I’ve set up works for the web service, but I can’t seem to get it to work for gRPC. I’m using the https://github.com/kubernetes/ingress-nginx ingress controller with a network load balancer on an AWS EKS cluster.

When I run:

kubectl port-forward temporaltest-frontend-fb6b79468-cznvc 7233:7233

and then in another window:

❯ grpcurl -plaintext localhost:7233 list 
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
temporal.api.workflowservice.v1.WorkflowService
temporal.server.api.adminservice.v1.AdminService

I’m able to see the services on port 7233. This would seem to indicate that the service itself is running without issue and that the issue is with the Ingress resource or the ingress controller resource.

Here’s what I see if I try to grpcurl my endpoint:

❯ grpcurl -plaintext ssokolin-test.useavenue.com:443 list
Failed to dial target host "ssokolin-test.useavenue.com:443": context deadline exceeded

Any help here would be very much appreciated!!

My Ingress resource manifest is below:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    ingressclass.kubernetes.io/is-default-class: 'true'
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'false'
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '60'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '60'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '60'
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    nginx.org/grpc-services: temporaltest-frontend
  labels:
    app.kubernetes.io/component: web
    app.kubernetes.io/instance: temporaltest
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: temporal
    app.kubernetes.io/part-of: temporal
    app.kubernetes.io/version: 1.14.0
    helm.sh/chart: temporal-0.14.0
  name: temporaltest-web
spec:
  rules:
    - host: ssokolin-test.useavenue.com
      http:
        paths:
          - backend:
              serviceName: temporaltest-frontend
              servicePort: 7233
            path: /
  tls:
    - hosts:
        - ssokolin-test.useavenue.com
      secretName: secret1

My Temporal frontend Service manifest is below:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: 'true'
  labels:
    app.kubernetes.io/component: frontend
    app.kubernetes.io/headless: 'true'
    app.kubernetes.io/instance: temporaltest
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: temporal
    app.kubernetes.io/part-of: temporal
    app.kubernetes.io/version: 1.14.0
    helm.sh/chart: temporal-0.14.0
    prometheus.io/job: temporal-frontend
    prometheus.io/port: '9090'
    prometheus.io/scheme: http
    prometheus.io/scrape: 'true'
  name: temporaltest-frontend-headless
spec:
  clusterIP: None
  ports:
    - name: grpc-rpc
      port: 7233
      protocol: TCP
      targetPort: rpc
    - name: metrics
      port: 9090
      protocol: TCP
      targetPort: metrics
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/component: frontend
    app.kubernetes.io/instance: temporaltest
    app.kubernetes.io/name: temporal
  type: ClusterIP

My Ingress controller Service manifest is below:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.43.0
    helm.sh/chart: ingress-nginx-3.20.1
  name: nginx-ingress-ingress-nginx-controller
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer
2

do you get the same message with authority header in this grpcurl command:
-authority

what about if you use ip directly?

3

Yeah I get the same message, both for authority and using ip directly

4

It seems like ingress nginx is expecting tls (evidence by port 443 and the tls secret on the K8s Ingress Object). Can you try grpcurl specifying the client cert and key?

5

Ah I realize you’re not trying to do mtls auth. Are there any logs in either nginx or front end pod when attempting to connect you could share? Can you try port forwarding to the service instead of directly to pod?

6

Port forwarding to the service works:

❯ kubectl port-forward service/temporaltest-frontend 7233:7233
Forwarding from 127.0.0.1:7233 -> 7233
Forwarding from [::1]:7233 -> 7233
Handling connection for 7233

❯ grpcurl -plaintext localhost:7233 list
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
temporal.api.workflowservice.v1.WorkflowService
temporal.server.api.adminservice.v1.AdminService
7

I got it!!

❯ grpcurl temporal-frontend.ssokolin-test.com:443 list
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
temporal.api.workflowservice.v1.WorkflowService
temporal.server.api.adminservice.v1.AdminService

I’m pretty sure the issue was that I was getting rate-limited by the TLS Certificate Authority

When I switched to their staging url, it worked:

1 Like
8

Thanks for the update! Best wishes!

9

Hi @Sam_Sokolin

I’m also getting same issue at local. I’m using minikube. Issue is exactly same. Service is working fine if I do port forwarding to the service or deployment but when I go via ingress, I also get same error.

Failed to dial target host “10.106.254.229:443”: context deadline exceeded

I’m running the below command

grpcurl -v -d ‘{“greeting”: “Kong Hello world!”}’ -insecure $PROXY_IP:443 hello.HelloService.SayHello

please help me with this… what you changed and how it worked for you.

1 Like
10

Hi @Sam_Sokolin Since for exposing frontend svc your ingress path is / so at which path are you exposing web svc? as even i am trying to expose frontend svc via nginx ingress and when exposing with path / its working fine but as soon as i give path as /frontend it starts giving protocol error… since its working with earlier means nginx supports grpc…now the issue is that even web ui is exposed under host / by giving anyother path like /web ui is not loading.