# Temporal server with the error "UpdateTaskQueue failed"

**URL:** <https://community.temporal.io/t/temporal-server-with-the-error-updatetaskqueue-failed/7813>\
**Category:** Community Support\
**Tags:** go-sdk, mysql\
**Created:** [April 7, 2023, 3:21am UTC](https://community.temporal.io/t/temporal-server-with-the-error-updatetaskqueue-failed/7813 "2023-04-07T03:21:15Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![zhangkyou](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/zhangkyou/32/3528_2.png) [@zhangkyou](https://community.temporal.io/u/zhangkyou)\
**Post date:** [April 7, 2023, 3:21am UTC](https://community.temporal.io/t/temporal-server-with-the-error-updatetaskqueue-failed/7813/1 "2023-04-07T03:21:15Z")

</div>

I found there is many error log in my Temporal server’s log.

```auto
{"level":"error","ts":"2023-04-07T10:36:11.662+0800","msg":"Operation failed with internal error.","error":"UpdateTaskQueue failed. Failed to start transaction. Error: invalid connection","operation":"UpdateTaskQueue","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/gaia/workspace-job/git.xiaojukeji.com/soda-server/temporal-server/internal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/gaia/workspace-job/git.xiaojukeji.com/soda-server/temporal-server/internal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/gaia/workspace-job/git.xiaojukeji.com/soda-server/temporal-server/internal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).UpdateTaskQueue.func1\n\t/gaia/workspace-job/git.xiaojukeji.com/soda-server/temporal-server/internal/common/persistence/persistenceMetricClients.go:555\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).UpdateTaskQueue\n\t/gaia/workspace-job/git.xiaojukeji.com/soda-server/temporal-server/internal/common/persistence/persistenceMetricClients.go:557\ngo.temporal.io/server/service/matching.(*taskQueueDB).updateTaskQueue\n\t/gaia/workspace-job/git.xiaojukeji.com/soda-server/temporal-server/internal/service/matching/db.go:376\ngo.temporal.io/server/service/matching.(*taskQueueDB).UpdateState\n\t/gaia/workspace-job/git.xiaojukeji.com/soda-server/temporal-server/internal/service/matching/db.go:186\ngo.temporal.io/server/service/matching.(*taskReader).persistAckLevel\n\t/gaia/workspace-job/git.xiaojukeji.com/soda-server/temporal-server/internal/service/matching/taskReader.go:297\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/gaia/workspace-job/git.xiaojukeji.com/soda-server/temporal-server/internal/service/matching/taskReader.go:209\ngo.temporal.io/server/internal/goro.(*Group).Go.func1\n\t/gaia/workspace-job/git.xiaojukeji.com/soda-server/temporal-server/internal/internal/goro/group.go:58"}

```

```auto
{"level":"error","ts":"2023-04-07T10:36:11.662+0800","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","store-operation":"update-task-queue","error":"UpdateTaskQueue failed. Failed to start transaction. Error: invalid connection","logging-call-at":"taskReader.go:212","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/gaia/workspace-job/git.xiaojukeji.com/soda-server/temporal-server/internal/common/log/zap_logger.go:150\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/gaia/workspace-job/git.xiaojukeji.com/soda-server/temporal-server/internal/service/matching/taskReader.go:212\ngo.temporal.io/server/internal/goro.(*Group).Go.func1\n\t/gaia/workspace-job/git.xiaojukeji.com/soda-server/temporal-server/internal/internal/goro/group.go:58"}

```

Temporal server version: 1.20.0  
Mysql version: v57

It seems that the error was reported from this part of the code.The `m.txExecute`

```auto
func (m *sqlTaskManager) UpdateTaskQueue(
	ctx context.Context,
	request *persistence.InternalUpdateTaskQueueRequest,
) (*persistence.UpdateTaskQueueResponse, error) {
	nidBytes, err := primitives.ParseUUID(request.NamespaceID)
	if err != nil {
		return nil, serviceerror.NewInternal(err.Error())
	}

	tqId, tqHash := m.taskQueueIdAndHash(nidBytes, request.TaskQueue, request.TaskType)
	var resp *persistence.UpdateTaskQueueResponse
	err = m.txExecute(ctx, "UpdateTaskQueue", func(tx sqlplugin.Tx) error {
		if err := lockTaskQueue(ctx,
			tx,
			tqHash,
			tqId,
			request.PrevRangeID,
		); err != nil {
			return err
		}
		result, err := tx.UpdateTaskQueues(ctx, &sqlplugin.TaskQueuesRow{
			RangeHash: tqHash,
			TaskQueueID: tqId,
			RangeID: request.RangeID,
			Data: request.TaskQueueInfo.Data,
			DataEncoding: request.TaskQueueInfo.EncodingType.String(),
		})
		if err != nil {
			return err
		}
		rowsAffected, err := result.RowsAffected()
		if err != nil {
			return err
		}
		if rowsAffected != 1 {
			return fmt.Errorf("%v rows were affected instead of 1", rowsAffected)
		}
		resp = &persistence.UpdateTaskQueueResponse{}
		return nil
	})
	return resp, err
}

```

The issue is that the sample workflow occasionally gets stuck at either the WorkflowTaskScheduled event or the ActivityTaskSchedule event, and remains there until the workflow eventually times out.

I’m not sure if this issue is related to the error logs mentioned above.

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [April 7, 2023, 3:58pm UTC](https://community.temporal.io/t/temporal-server-with-the-error-updatetaskqueue-failed/7813/2 "2023-04-07T15:58:29Z")

</div>

> Error: invalid connection

Looks like issues connecting to your persistence store. Does this only happen with matching hosts? Check your matching hosts static configs (/etc/temporal/config/docker.yaml).

Maybe also worth checking persistence latencies:

`histogram_quantile(0.95, sum(rate(persistence_latency_bucket{}[1m])) by (operation, le))`

and resource exhausted on SystemOverloaded cause:

```auto
sum(rate(service_errors_resource_exhausted{}[1m])) by (operation, resource_exhausted_cause)

```

---

<div class="post-metadata">

**Author:** ![zhangkyou](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/zhangkyou/32/3528_2.png) [@zhangkyou](https://community.temporal.io/u/zhangkyou)\
**Post date:** [April 10, 2023, 2:44am UTC](https://community.temporal.io/t/temporal-server-with-the-error-updatetaskqueue-failed/7813/3 "2023-04-10T02:44:10Z")

</div>

> [@tihomir](#):
>
> `service_errors_resource_exhausted`

The metric persistence\_latency\_bucket appears to be normal.

 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/4/4307724db1784aa0f334eea9eded5a8d5189ff17.png)

And the metric service\_errors\_resource\_exhausted didn’t emit.

---

<div class="post-metadata">

**Author:** ![zhangkyou](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/zhangkyou/32/3528_2.png) [@zhangkyou](https://community.temporal.io/u/zhangkyou)\
**Post date:** [April 10, 2023, 7:04am UTC](https://community.temporal.io/t/temporal-server-with-the-error-updatetaskqueue-failed/7813/4 "2023-04-10T07:04:49Z")

</div>

> [@zhangkyou](#):
>
> `m.txExecute`

We are using a DBProxy in front of our MySQL database. Could you please tell me what SQL statements are used in the `m.txExecute` method? I want to check if it’s because our DBProxy does not support them.

---

<div class="post-metadata">

**Author:** ![zhangkyou](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/zhangkyou/32/3528_2.png) [@zhangkyou](https://community.temporal.io/u/zhangkyou)\
**Post date:** [April 10, 2023, 7:50am UTC](https://community.temporal.io/t/temporal-server-with-the-error-updatetaskqueue-failed/7813/5 "2023-04-10T07:50:43Z")

</div>

Yes，it only happen with matching service.My frontend, matching, history, worker services use the same static config. I start the service with the cmd

```auto
temporal-server --config="config/" start --services="frontend,matching,history,worker"

```

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [April 14, 2023, 2:34pm UTC](https://community.temporal.io/t/temporal-server-with-the-error-updatetaskqueue-failed/7813/6 "2023-04-14T14:34:51Z")

</div>

Any relevant errors in your dbproxy logs?  
The errors you shared point to connection issues, “Error: invalid connection”.

> The issue is that the sample workflow occasionally gets stuck at either the WorkflowTaskScheduled event or the ActivityTaskSchedule event, and remains there until the workflow eventually times out.

Also if you can share event history of one of those “stuck” executions please.

---

<div class="post-metadata">

**Author:** ![zhangkyou](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/zhangkyou/32/3528_2.png) [@zhangkyou](https://community.temporal.io/u/zhangkyou)\
**Post date:** [April 17, 2023, 3:06am UTC](https://community.temporal.io/t/temporal-server-with-the-error-updatetaskqueue-failed/7813/7 "2023-04-17T03:06:32Z")

</div>

There isn’t any log in dbproxy.

And there is a stuck workflow that hang at WorkflowTaskScheduled event.

```auto
[
  {
    "eventId": "1",
    "eventTime": "2023-04-16T11:49:00.299533562Z",
    "eventType": "WorkflowExecutionStarted",
    "version": "0",
    "taskId": "4468293",
    "workerMayIgnore": false,
    "workflowExecutionStartedEventAttributes": {
      "workflowType": {
        "name": "FeaturePaletteProducerWorkflow"
      },
      "parentWorkflowNamespace": "",
      "parentWorkflowNamespaceId": "",
      "parentWorkflowExecution": null,
      "parentInitiatedEventId": "0",
      "taskQueue": {
        "name": "feature-palette",
        "kind": "Normal"
      },
      "input": {
        "payloads": [
          {
            "metadata": {
              "encoding": "anNvbi9wbGFpbg=="
            },
            "data": "eyJmZWF0dXJlX25hbWUiOiJhcmVhLkRpc3BhdGNoLmRpc3BhdGNoX3N0b2NrZWQiLCJzcWwiOiJzZWxlY3QgY2l0eV9pZCBhcyBjaXR5X2lkLCBmcm9tX2FyZWFfaWQgYXMgb2JqZWN0X2lkLCAnYXJlYTpEaXNwYXRjaDpkaXNwYXRjaF9zdG9ja2VkJyBhcyBmZWF0dXJlX25hbWUsIHN1bShjYXNlIHdoZW4gZmlyc3Rfc2NvcmVfdHMgPSAwIHRoZW4gMSBlbHNlIDAgZW5kKSAgYXMgZmVhdHVyZV92YWx1ZSBmcm9tIGRfZGVsaXZlcnlfYWxsICAgIHdoZXJlIChjcmVhdGVfdGltZSBcdTAwM2UgVU5JWF9USU1FU1RBTVAoKSAtIDYwMCBhbmQgZnJvbV9hcmVhX2lkIFx1MDAzZSAwKSAgICBncm91cCBieSBjaXR5X2lkLCBmcm9tX2FyZWFfaWQiLCJkYXRhX3NvdXJjZSI6ImRlbGl2ZXJ5X29yZGVyX3dpZGVfdGFibGUiLCJjcm9uX3N0ciI6IiogKiAqICogKiIsInRpbWVvdXQiOjUwMDAwMDAwMDAwfQ=="
          }
        ]
      },
      "workflowExecutionTimeout": "0s",
      "workflowRunTimeout": "50s",
      "workflowTaskTimeout": "10s",
      "continuedExecutionRunId": "b6f741c8-325c-464b-860c-74ce58352091",
      "initiator": "CronSchedule",
      "continuedFailure": null,
      "lastCompletionResult": {
        "payloads": [
          {
            "metadata": {
              "encoding": "anNvbi9wbGFpbg=="
            },
            "data": "InN1Y2Nlc3Mi"
          }
        ]
      },
      "originalExecutionRunId": "5872de82-3541-4d47-b11e-e49168987c10",
      "identity": "",
      "firstExecutionRunId": "9546ae2b-cbbd-407d-95ac-f88732c20a99",
      "retryPolicy": {
        "initialInterval": "1s",
        "backoffCoefficient": 2,
        "maximumInterval": "20s",
        "maximumAttempts": 3,
        "nonRetryableErrorTypes": []
      },
      "attempt": 1,
      "workflowExecutionExpirationTime": null,
      "cronSchedule": "* * * * *",
      "firstWorkflowTaskBackoff": "60s",
      "memo": null,
      "searchAttributes": null,
      "prevAutoResetPoints": {
        "points": [
          {
            "binaryChecksum": "e09e4f7f586b102723272ca02a06a1ed",
            "runId": "9546ae2b-cbbd-407d-95ac-f88732c20a99",
            "firstWorkflowTaskCompletedId": "4",
            "createTime": "2023-04-11T03:05:00.820153573Z",
            "expireTime": "2023-04-14T03:05:00.901427868Z",
            "resettable": true
          },
          {
            "binaryChecksum": "76b3d3238b8dcf11c2cb3c82cf2903d0",
            "runId": "82449981-2ca9-42b0-9068-ea28fa6ed570",
            "firstWorkflowTaskCompletedId": "4",
            "createTime": "2023-04-13T07:36:00.695644487Z",
            "expireTime": "2023-04-16T07:36:00.749381633Z",
            "resettable": true
          }
        ]
      },
      "header": {
        "fields": {}
      },
      "parentInitiatedEventVersion": "0"
    }
  },
  {
    "eventId": "2",
    "eventTime": "2023-04-16T11:50:00.301273463Z",
    "eventType": "WorkflowTaskScheduled",
    "version": "0",
    "taskId": "4468303",
    "workerMayIgnore": false,
    "workflowTaskScheduledEventAttributes": {
      "taskQueue": {
        "name": "feature-palette",
        "kind": "Normal"
      },
      "startToCloseTimeout": "10s",
      "attempt": 1
    }
  },
  {
    "eventId": "3",
    "eventTime": "2023-04-16T11:50:50.302433608Z",
    "eventType": "WorkflowExecutionTimedOut",
    "version": "0",
    "taskId": "4468307",
    "workerMayIgnore": false,
    "workflowExecutionTimedOutEventAttributes": {
      "retryState": "InProgress",
      "newExecutionRunId": "8bc15396-163b-42d9-9e3d-e0124cf8cf33"
    }
  }
]

```

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [April 17, 2023, 3:30am UTC](https://community.temporal.io/t/temporal-server-with-the-error-updatetaskqueue-failed/7813/8 "2023-04-17T03:30:41Z")

</div>

Do you have workers polling on the `feature-palette` task queue?  
Could check with cli or tctl, for example:

```auto
temporal task-queue describe -t feature-palette

tctl tq desc -tq feature-palette

```

What’s the reason to set workflow run timeout for this cron execution to 50s?  
Service in this case correctly scheduled the first workflow task and timed out the execution when workflow run timeout was reached, so not sure if this might be related to original issue. Would check if you have workers polling. If you do maybe check service errors:

`sum(rate(service_error_with_type{service_type="frontend"}[5m])) by (error_type)`

and your workers workflow task schedule to start latencies:

`histogram_quantile(0.95, sum(rate(temporal_workflow_task_schedule_to_start_latency_bucket[5m])) by (namespace, le))`

---

<div class="post-metadata">

**Author:** ![zhangkyou](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/zhangkyou/32/3528_2.png) [@zhangkyou](https://community.temporal.io/u/zhangkyou)\
**Post date:** [April 20, 2023, 7:55am UTC](https://community.temporal.io/t/temporal-server-with-the-error-updatetaskqueue-failed/7813/9 "2023-04-20T07:55:44Z")

</div>

1.There are three workers polling on the feature-palette. If there are no workers available in the feature-palette task queue, all the workflow run instances on the feature-palette will timeout. However, some workflow run instances occasionally encounter this problem.  
  
2.I need to run this workflow every minute, so each run instance of the workflow must be completed within 50 seconds in order not to affect the execution of the next minute. WorkflowTaskScheduled is a very quick process and should not take 50 seconds to complete. My Temporal cluster is only running two workflows.  
  
3.As for the issue with service errors, there are some errors in the namespace I’m using, but the time when the errors occurred does not match the time when the WorkflowTaskScheduled was hung up.

 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/a/a1d898b1954824c93607b542e79bbc657d034751.png)  
  
4. The metric `temporal_workflow_task_schedule_to_start_latency_bucket` only contains data from the namespace `temporal_system` . There were only two data points recorded for this metric in one day.  
 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/4/4cabf1a2cc1e41c9db019a5c521d773f35488576.png)

---

<div class="post-metadata">

**Author:** ![sai\_liu](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/sai_liu/32/7149_2.png) [@sai\_liu](https://community.temporal.io/u/sai_liu)\
**Post date:** [September 13, 2025, 2:39am UTC](https://community.temporal.io/t/temporal-server-with-the-error-updatetaskqueue-failed/7813/10 "2025-09-13T02:39:43Z")

</div>

@zhangkyou @tihomir does this problem solved？ meet same issue.

---

<div class="post-metadata">

**Author:** ![Roger\_Demetrescu](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/roger_demetrescu/32/8303_2.png) [@Roger\_Demetrescu](https://community.temporal.io/u/Roger_Demetrescu)\
**Post date:** [April 10, 2026, 12:50am UTC](https://community.temporal.io/t/temporal-server-with-the-error-updatetaskqueue-failed/7813/11 "2026-04-10T00:50:57Z")

</div>

I`m having the same issue… lots of messages like:

time=2026-04-09T21:45:23.567 level=ERROR msg=“Operation failed with internal error.” error=“UpdateTaskQueue failed. Failed to start transaction. Error: context canceled” error-type=serviceerror.Unavailable operation=UpdateTaskQueue

time=2026-04-09T21:45:23.567 level=ERROR msg=“Persistent store operation failure” component=matching-engine wf-task-queue-name=/\_sys/default-worker-tq/3 wf-task-queue-type=Workflow wf-namespace=temporal-system worker-version=\_unversioned\_ backlog=classic store-operation=update-task-queue error=“UpdateTaskQueue failed. Failed to start transaction. Error: context canceled”

Important detail: I’m running temporal without any kind of persistence, on my machine:

**temporal server start-dev**