Ok, i got replication setup working, my key learnings were
a) its the DR( replication site) which pulls the data from main site, so if you are setting up VPC… its important for your secondary region to reach the primary region, and not vice versa.
b) as @arnesenfamily mentioned, L7 load balncers does not work (ALB, NGIIX ETC). so L4 NLB is your friend.
c) @arnesenfamily observation about Helm is correct. you may have to modify your helms to expose your front end as an ingress/service so that it could be consumed externally.
I used this yaml for exposing my front end through an internal nlb
apiVersion: v1
kind: Service
metadata:
name: temporal-dr
namespace: temporal
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-type: nlb
spec:
type: LoadBalancer
ports:
- name: grpc-rpc
port: 7233
protocol: TCP
targetPort: rpc
selector:
app.kubernetes.io/component: frontend
app.kubernetes.io/instance: temporal
app.kubernetes.io/name: temporal