It sometimes takes too long to register workflow

Community Support
,
1

We’ve added a timeout of 100 secs, but it still timeout. sometimes it creates in a second, sometimes it’s stuck.

2020-10-23T16:37:19.304026172Z  2020-10-23T16:37:19.303Z	ERROR	common/helper.go:101	Failed to create workflow	{"error": "code:deadline-exceeded message:timeout"}

we’re using the go SDK like this:

we, err := workflowClient.StartWorkflow(ctx, options, workflow, args...)

We’re using cadence v0.12.0 with Cassandra.

2

Can you reproduce it using Temporal?

3

I actually fixed it. There was probably a network connectivity problem with one of my nodes. I deleted all Cadence replicasets, and then couldn’t replicate this problem anymore.

I’ve added short retries in the cadence clientside to ensure that if one of the services has a communication problem, load balancing can ensure that a healthy pod is able to fulfill the request.

I don’t know if something like LinkerD would help in these situations in production. Don’t currently use it.

1 Like