Exposing grpc Frontend via ALB

Community Support
, , ,
1

Hi There

Currently in the process of deploying out a temporal environment and have hit a roadblock in terms of how to successfully expose the Frontend service. With the service showing as available from the LB but unable to connect using grpcurl.

Current configuration :

  • DNS Name and ALB are provisioned externally via terraform
  • Terraform deploys the Temporal Helm chart with configuration setup to expose the Frontend service to a defined Nodeport
  • preconfigured Loadbalancer connects to the TG over 443 and from then the target group sends traffic onto the given Nodeport
  • Health Check is passing on the LB when testing on endpoint /grpc.health.v1.Health/Check"

Using the following config to setup the targetgroup mapping with the Node Port

{{ if .Values.server.enabled }}
apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: {{ include "temporal.componentname" (list . "frontend") }}
spec:
  serviceRef:
    name: {{ include "temporal.componentname" (list . "frontend") }}
    port: {{ .Values.server.frontend.service.port }}
  targetGroupARN: {{ .Values.server.frontend.service.targetGroupARN }}
{{- end }}

Screenshot 2022-06-17 at 16.59.24
Screenshot 2022-06-17 at 16.59.244424×762 124 KB 

grpcurl -vv temporal-frontend.domain.net:443 list
Failed to list services: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway)

Any help is appreciated !

2 
grpcurl -plaintext 127.0.0.1:7233 list
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
temporal.api.operatorservice.v1.OperatorService
temporal.api.workflowservice.v1.WorkflowService
temporal.server.api.adminservice.v1.AdminServ

Local port forwarding works

3

To health check Temporal services you can use grpc-health-probe, for example:

Frontend:
./grpc-health-probe -addr=ADDRESS:PORT -service=temporal.api.workflowservice.v1.WorkflowService

(you can also use cli for this):
tctl --address localhost:7233 cluster health

Matching service:
./grpc-health-probe -addr=ADDRESS:PORT -service=temporal.api.workflowservice.v1.MatchingService

History service:
./grpc-health-probe -addr=ADDRESS:PORT -service=temporal.api.workflowservice.v1.HistoryService

Worker service does not currently expose a health check endpoint.

For LB, do you have SSL pass through enabled? Not AWS expert but it should be able to receive gRPC requests and forward them to your frontend service(s) (7233 default port).

4

Looks like ALB doesnt support SSL pass through.

Testing out creating a NLB for the frontend traffic.

5

Changed to using a NLB with TCP to go via

  • nlb:7233 > Allocated K8s: frontend:nodeport > frontend:7233

This allowed the connection through for the POC.

Up next is trying to setup mTLS for the frontend :grimacing: