# Struggling to get ingress working for frontend gRPC service

**URL:** <https://community.temporal.io/t/struggling-to-get-ingress-working-for-frontend-grpc-service/4018>\
**Category:** Community Support\
**Tags:** kubernetes, grpc\
**Created:** [February 17, 2022, 3:45pm UTC](https://community.temporal.io/t/struggling-to-get-ingress-working-for-frontend-grpc-service/4018 "2022-02-17T15:45:54Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sam\_Sokolin](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/sam_sokolin/32/1759_2.png) [@Sam\_Sokolin](https://community.temporal.io/u/Sam_Sokolin)\
**Post date:** [February 17, 2022, 3:45pm UTC](https://community.temporal.io/t/struggling-to-get-ingress-working-for-frontend-grpc-service/4018/1 "2022-02-17T15:45:54Z")

</div>

I’ve deployed Temporal on K8s and I’m not able to connect to the frontend gRPC service from outside the cluster. The ingress I’ve set up works for the web service, but I can’t seem to get it to work for gRPC. I’m using the [https://github.com/kubernetes/ingress-nginx](https://kubernetes/ingress-nginx) ingress controller with a network load balancer on an AWS EKS cluster.

When I run:

```auto
kubectl port-forward temporaltest-frontend-fb6b79468-cznvc 7233:7233

```

and then in another window:

```auto
❯ grpcurl -plaintext localhost:7233 list 
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
temporal.api.workflowservice.v1.WorkflowService
temporal.server.api.adminservice.v1.AdminService

```

I’m able to see the services on port 7233. This would seem to indicate that the service itself is running without issue and that the issue is with the Ingress resource or the ingress controller resource.

Here’s what I see if I try to `grpcurl` my endpoint:

```auto
❯ grpcurl -plaintext ssokolin-test.useavenue.com:443 list
Failed to dial target host "ssokolin-test.useavenue.com:443": context deadline exceeded

```

Any help here would be very much appreciated!!

My Ingress resource manifest is below:

```auto
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    ingressclass.kubernetes.io/is-default-class: 'true'
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'false'
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '60'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '60'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '60'
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    nginx.org/grpc-services: temporaltest-frontend
  labels:
    app.kubernetes.io/component: web
    app.kubernetes.io/instance: temporaltest
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: temporal
    app.kubernetes.io/part-of: temporal
    app.kubernetes.io/version: 1.14.0
    helm.sh/chart: temporal-0.14.0
  name: temporaltest-web
spec:
  rules:
    - host: ssokolin-test.useavenue.com
      http:
        paths:
          - backend:
              serviceName: temporaltest-frontend
              servicePort: 7233
            path: /
  tls:
    - hosts:
        - ssokolin-test.useavenue.com
      secretName: secret1

```

My Temporal frontend Service manifest is below:

```auto
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: 'true'
  labels:
    app.kubernetes.io/component: frontend
    app.kubernetes.io/headless: 'true'
    app.kubernetes.io/instance: temporaltest
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: temporal
    app.kubernetes.io/part-of: temporal
    app.kubernetes.io/version: 1.14.0
    helm.sh/chart: temporal-0.14.0
    prometheus.io/job: temporal-frontend
    prometheus.io/port: '9090'
    prometheus.io/scheme: http
    prometheus.io/scrape: 'true'
  name: temporaltest-frontend-headless
spec:
  clusterIP: None
  ports:
    - name: grpc-rpc
      port: 7233
      protocol: TCP
      targetPort: rpc
    - name: metrics
      port: 9090
      protocol: TCP
      targetPort: metrics
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/component: frontend
    app.kubernetes.io/instance: temporaltest
    app.kubernetes.io/name: temporal
  type: ClusterIP

```

My Ingress controller Service manifest is below:

```auto
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.43.0
    helm.sh/chart: ingress-nginx-3.20.1
  name: nginx-ingress-ingress-nginx-controller
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer

```

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [February 17, 2022, 7:54pm UTC](https://community.temporal.io/t/struggling-to-get-ingress-working-for-frontend-grpc-service/4018/2 "2022-02-17T19:54:23Z")

</div>

> [@Sam\_Sokolin](#):
>
> ```auto
> ❯ grpcurl -plaintext ssokolin-test.useavenue.com:443 list
> Failed to dial target host "ssokolin-test.useavenue.com:443": context deadline exceeded
> 
> ```

do you get the same message with authority header in this grpcurl command:  
`-authority`

what about if you use ip directly?

---

<div class="post-metadata">

**Author:** ![Sam\_Sokolin](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/sam_sokolin/32/1759_2.png) [@Sam\_Sokolin](https://community.temporal.io/u/Sam_Sokolin)\
**Post date:** [February 18, 2022, 3:06am UTC](https://community.temporal.io/t/struggling-to-get-ingress-working-for-frontend-grpc-service/4018/3 "2022-02-18T03:06:21Z")

</div>

Yeah I get the same message, both for authority and using ip directly

---

<div class="post-metadata">

**Author:** ![joeandaverde](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/joeandaverde/32/1078_2.png) [@joeandaverde](https://community.temporal.io/u/joeandaverde)\
**Post date:** [February 18, 2022, 4:44am UTC](https://community.temporal.io/t/struggling-to-get-ingress-working-for-frontend-grpc-service/4018/4 "2022-02-18T04:44:14Z")

</div>

It seems like ingress nginx is expecting tls (evidence by port 443 and the tls secret on the K8s Ingress Object). Can you try grpcurl specifying the client cert and key?

---

<div class="post-metadata">

**Author:** ![joeandaverde](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/joeandaverde/32/1078_2.png) [@joeandaverde](https://community.temporal.io/u/joeandaverde)\
**Post date:** [February 18, 2022, 5:06am UTC](https://community.temporal.io/t/struggling-to-get-ingress-working-for-frontend-grpc-service/4018/5 "2022-02-18T05:06:23Z")

</div>

Ah I realize you’re not trying to do mtls auth. Are there any logs in either nginx or front end pod when attempting to connect you could share? Can you try port forwarding to the service instead of directly to pod?

---

<div class="post-metadata">

**Author:** ![Sam\_Sokolin](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/sam_sokolin/32/1759_2.png) [@Sam\_Sokolin](https://community.temporal.io/u/Sam_Sokolin)\
**Post date:** [February 18, 2022, 4:41pm UTC](https://community.temporal.io/t/struggling-to-get-ingress-working-for-frontend-grpc-service/4018/6 "2022-02-18T16:41:02Z")

</div>

Port forwarding to the service works:

```auto
❯ kubectl port-forward service/temporaltest-frontend 7233:7233
Forwarding from 127.0.0.1:7233 -> 7233
Forwarding from [::1]:7233 -> 7233
Handling connection for 7233

```

```auto
❯ grpcurl -plaintext localhost:7233 list
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
temporal.api.workflowservice.v1.WorkflowService
temporal.server.api.adminservice.v1.AdminService

```

---

<div class="post-metadata">

**Author:** ![Sam\_Sokolin](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/sam_sokolin/32/1759_2.png) [@Sam\_Sokolin](https://community.temporal.io/u/Sam_Sokolin)\
**Post date:** [February 18, 2022, 6:32pm UTC](https://community.temporal.io/t/struggling-to-get-ingress-working-for-frontend-grpc-service/4018/7 "2022-02-18T18:32:40Z")

</div>

I got it!!

```auto
❯ grpcurl temporal-frontend.ssokolin-test.com:443 list
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
temporal.api.workflowservice.v1.WorkflowService
temporal.server.api.adminservice.v1.AdminService

```

I’m pretty sure the issue was that I was getting rate-limited by the TLS Certificate Authority

> **[Rate Limits - Let's Encrypt](https://letsencrypt.org/docs/rate-limits/)**
>
> Let’s Encrypt provides rate limits to ensure fair usage by as many people as possible. We believe these rate limits are high enough to work for most people by default. We’ve also designed them so renewing a certificate almost never hits a...

When I switched to their staging url, it worked:

> **[Staging Environment - Let's Encrypt](https://letsencrypt.org/docs/staging-environment/)**
>
> We highly recommend testing against our staging environment before using our production environment. This will allow you to get things right before issuing trusted certificates and reduce the chance of your running up against rate limits.
> The ACME...

---

<div class="post-metadata">

**Author:** ![joeandaverde](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/joeandaverde/32/1078_2.png) [@joeandaverde](https://community.temporal.io/u/joeandaverde)\
**Post date:** [February 18, 2022, 9:15pm UTC](https://community.temporal.io/t/struggling-to-get-ingress-working-for-frontend-grpc-service/4018/8 "2022-02-18T21:15:55Z")

</div>

Thanks for the update! Best wishes!

---

<div class="post-metadata">

**Author:** ![Saurabh\_Sharma](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/saurabh_sharma/32/3163_2.png) [@Saurabh\_Sharma](https://community.temporal.io/u/Saurabh_Sharma)\
**Post date:** [January 11, 2023, 5:25pm UTC](https://community.temporal.io/t/struggling-to-get-ingress-working-for-frontend-grpc-service/4018/9 "2023-01-11T17:25:46Z")

</div>

Hi @Sam_Sokolin

I’m also getting same issue at local. I’m using minikube. Issue is exactly same. Service is working fine if I do port forwarding to the service or deployment but when I go via ingress, I also get same error.

Failed to dial target host “10.106.254.229:443”: context deadline exceeded

I’m running the below command

**grpcurl -v -d ‘{“greeting”: “Kong Hello world!”}’ -insecure $PROXY\_IP:443 hello.HelloService.SayHello**

please help me with this… what you changed and how it worked for you.

---

<div class="post-metadata">

**Author:** ![Sameer\_Kumar](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/sameer_kumar/32/5339_2.png) [@Sameer\_Kumar](https://community.temporal.io/u/Sameer_Kumar)\
**Post date:** [May 14, 2024, 11:34am UTC](https://community.temporal.io/t/struggling-to-get-ingress-working-for-frontend-grpc-service/4018/10 "2024-05-14T11:34:45Z")

</div>

Hi @Sam_Sokolin Since for exposing frontend svc your ingress path is / so at which path are you exposing web svc? as even i am trying to expose frontend svc via nginx ingress and when exposing with path / its working fine but as soon as i give path as /frontend it starts giving protocol error… since its working with earlier means nginx supports grpc…now the issue is that even web ui is exposed under host / by giving anyother path like /web ui is not loading.