Client doesn't connect: timed out waiting for server handshake

I have temporal 0.30.0 deployed on Kubernetes, together with my activity & workflow workers. I have a custom Go CLI app, with sdk 0.30.0, which I use to test starting my workflows. I port-forward the frontend pod to my laptop and it is accessible on localhost:7233.

Here is the snippet to connect the CLI app to Temporal:

	c, err := client.NewClient(client.Options{
		HostPort: serviceInfo.Host + ":" + serviceInfo.Port,
		Logger:   logger,
	})
	if err != nil {
		zapLogger.Error("Unable to create client", zap.Error(err))
		return
	}

But when running the testing CLI app, I get this error printed by the zapLogger:

{"level":"error","ts":1601901106.624567,"caller":"app/temporal.go:45","msg":"Unable to create client","error":"health check error: last connection error: timed out waiting for server handshake","stacktrace":...}

What could be the problem here?

I also port-forwarded the web pod and I can access that one correctly using http://localhost:8080. Given this works, I assume the port-forwarding of the frontend service should also work correctly.

Have you tried using tctl? Does that work?

I tried a number of commands with tctl via the admintools container but always got this error:

Error: Must provide service configuration dir path.

Didn’t know where to look to get this resolved…

Which tctl command are you running? Try the non admin commands.

Argh… the order of these arguments is really confusing! I would rather use tctl namespace describe default.

$ tctl --namespace default namespace describe
Name: default
Id: ffe1aaaf-cc6e-4d23-9b09-c9cfc803e0be
Description: 
OwnerEmail: 
NamespaceData: map[string]string(nil)
State: Registered
RetentionInDays: 72h0m0s
ActiveClusterName: active
Clusters: active
HistoryArchivalState: Disabled
VisibilityArchivalState: Disabled
Bad binaries to reset:
+-----------------+----------+------------+--------+
| BINARY CHECKSUM | OPERATOR | START TIME | REASON |
+-----------------+----------+------------+--------+
+-----------------+----------+------------+--------+

This seems to be working. Looks like temporal-frontend is up and serving requests.

v0.30.0 had a bug where I override ConnectParmas but forgot to set MinConnectTimeout. This parameter is used by health checker (or any other client) as dial duration. I guess the default (if it is not set) is 0 and with port forwarding is enabled it just takes a little bit longer to connect, so health check times out.
This bug was fixed with 250 (set default back to gRPC default – 20 seconds). You can upgrade GoSDK to any next version (I would suggest v1.0.0) and it should work fine. Note: you don’t have to upgrade server (although I would suggest, of course).