# Workflow task timed out on GKE

**URL:** <https://community.temporal.io/t/workflow-task-timed-out-on-gke/4909>\
**Category:** Community Support\
**Tags:** java-sdk, cassandra, metrics\
**Created:** [June 7, 2022, 3:14pm UTC](https://community.temporal.io/t/workflow-task-timed-out-on-gke/4909 "2022-06-07T15:14:47Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [June 8, 2022, 4:18am UTC](https://community.temporal.io/t/workflow-task-timed-out-on-gke/4909/3 "2022-06-08T04:18:38Z")

</div>

Could you provide full history for your execution:

`tctl wf show -w <wfid> -r <runid> --output_filename myhistory.json`

Did you have the chance look through the [worker tuning guide in docs](https://docs.temporal.io/operation/how-to-tune-workers/)?

Could you provide info on your sync match rate:

```auto
sum(rate(poll_success_sync{}[1m])) / sum(rate(poll_success{}[1m]))

```

ideally it should be above 99%. if sync match rate is low it would mean your workers are unable to keep up (need to increase worker capacity)

another thing to look at is sdk `task_schedule_to_start_latency` metric, can you measure this latency as well? a high latency would indicate to add more workers.

> 1. NoHistoryShards changed to 32

i think this is too low, typically you would go with 512 for a small scale setup. for prod setup would start with 4K.

Another thing to look at are persistence latencies:

`histogram_quantile(0.95, sum(rate(persistence_latency_bucket{}[1m])) by (operation, le))`  
for operations: `CreateWorkflowExecution`, `UpdateWorkflowExecution`, `UpdateShard`

> Following is my temporal setup on GKE single node cluster.

How man instances of temporal services are you running on your test env? See [here](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/3) for recommendations for a prod setup

---

_[View the full topic](https://community.temporal.io/t/workflow-task-timed-out-on-gke/4909)._