Client doesn't connect: timed out waiting for server handshake

Community Support
1

I have temporal 0.30.0 deployed on Kubernetes, together with my activity & workflow workers. I have a custom Go CLI app, with sdk 0.30.0, which I use to test starting my workflows. I port-forward the frontend pod to my laptop and it is accessible on localhost:7233.

Here is the snippet to connect the CLI app to Temporal:

	c, err := client.NewClient(client.Options{
		HostPort: serviceInfo.Host + ":" + serviceInfo.Port,
		Logger:   logger,
	})
	if err != nil {
		zapLogger.Error("Unable to create client", zap.Error(err))
		return
	}

But when running the testing CLI app, I get this error printed by the zapLogger:

{"level":"error","ts":1601901106.624567,"caller":"app/temporal.go:45","msg":"Unable to create client","error":"health check error: last connection error: timed out waiting for server handshake","stacktrace":...}

What could be the problem here?

I also port-forwarded the web pod and I can access that one correctly using http://localhost:8080. Given this works, I assume the port-forwarding of the frontend service should also work correctly.

2

Have you tried using tctl? Does that work?

3

I tried a number of commands with tctl via the admintools container but always got this error:

Error: Must provide service configuration dir path.

Didn’t know where to look to get this resolved…

4

Which tctl command are you running? Try the non admin commands.

5

Argh… the order of these arguments is really confusing! I would rather use tctl namespace describe default.

$ tctl --namespace default namespace describe
Name: default
Id: ffe1aaaf-cc6e-4d23-9b09-c9cfc803e0be
Description: 
OwnerEmail: 
NamespaceData: map[string]string(nil)
State: Registered
RetentionInDays: 72h0m0s
ActiveClusterName: active
Clusters: active
HistoryArchivalState: Disabled
VisibilityArchivalState: Disabled
Bad binaries to reset:
+-----------------+----------+------------+--------+
| BINARY CHECKSUM | OPERATOR | START TIME | REASON |
+-----------------+----------+------------+--------+
+-----------------+----------+------------+--------+
6

This seems to be working. Looks like temporal-frontend is up and serving requests.

7

v0.30.0 had a bug where I override ConnectParmas but forgot to set MinConnectTimeout. This parameter is used by health checker (or any other client) as dial duration. I guess the default (if it is not set) is 0 and with port forwarding is enabled it just takes a little bit longer to connect, so health check times out.
This bug was fixed with 250 (set default back to gRPC default – 20 seconds). You can upgrade GoSDK to any next version (I would suggest v1.0.0) and it should work fine. Note: you don’t have to upgrade server (although I would suggest, of course).