Context deadline exceeded issue

Hi Team,
I am seeing context deadline exceeded error while writing the message to the queue (while scheduling the workflow). Can someone suggest any solution to resolve this issue ?
Any help is appreciated .


Hi @Mohit_Sharma can you provide more info please?
Full error log and a detailed description of what client code you are running would help.

	logger.BootstrapLogger.Debug("Entering initTemporalStarter...")
	// Create the client object just once per process
	opts := client.Options{HostPort: temporalHostPort, Namespace: temporalNameSpace}
	// Create the client object just once per process
	c, err := client.NewClient(opts)
	if err != nil {
		return nil, fmt.Errorf("error while creating the temporal client:%v", err)
	return c, nil

Above one is the client connection code.

options := client.StartWorkflowOptions{
ID: schedule.ID,
TaskQueue: worker.LIVETaskQueue,
_, err := s.Client.ExecuteWorkflow(context.Background(), options, “CreateWorkflow”, schedule)
if err != nil {
return fmt.Errorf(“unable to complete Workflow:%v”, err)

this the workflow schedule code.

time="2022-07-11 16:43:30" level=debug msg="Entering handler.buildGetSchemaFailureRespBody()[X-Tracking-Id:fc0c5b69-8324-40be-960d-ee2094070499]"```

this is the app log and the framework log is 
```Workflow:context deadline exceeded```

Thanks for the info, can you check if your temporal frontend service is up ?

tctl --ad <temporal-frontend-address:port> cl h

Does this return “Serving”? Use the same for temporal-frontend-address:port as you set in code, re: HostPort: temporalHostPort.

bash-5.1# tctl --ad cl h
temporal.api.workflowservice.v1.WorkflowService: SERVING

yes it is returning SERVING

Thanks. Would you be able to provide the whole error log?
Are you able to start your workflow execution via tctl for example:

tctl ---ad <temporal-frontend-address:port> --namespace <namespace> wf start --tq <taskqueue> --wt CreateWorkflow

what is the “schedule” argument you are passing as input arg?

Can you check your temporal service logs to see if anything stands out? These context deadline exceeded errors do need some debugging/are not easy to figure out at times.

bash-5.1# tctl --ad --namespace live-ctrl-svc wf start --tq LIVE_TASK_QUEUE --wt CreateWorkflow
Error: Failed to create workflow.
Error Details: context deadline exceeded
Stack trace:
goroutine 1 [running]:
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
        /usr/local/go/src/runtime/debug/stack.go:16 +0x19{0x1dbb50b, 0x1a}, {0x20dbf20, 0xc0000ac120})```

@tihomir Can you please have a look ?

Context deadline exceeded errors can happen for a number of reasons, for example network issues, config issues, services being down, db issues etc, so would need more info please:

How are you deploying temporal server (docker compose, helm charts, some other way)?
Do you see any logs reported by either temporal frontend, history, matching services. Do you see any logs reported by your db?
Do you have a load balancer in front of your temporal frontend service?
Can you access the web ui? If you can does it show any errors?

Any other info that you could share that you think would be important? Were you able to start workflows before or is this a fresh cluster install?