Updating existing/old schedules breaks the ability to run the workflow from the schedule

Community Support
1

Hello Temporal community! My team ran into an issue yesterday with the execution of a scheduled workflow. The workflow errored with:

unable to decode: json: cannot unmarshal object into Go value of type string

Normally, we would assume this is an issue with the input provided to the workflow, but after some investigation/troubleshooting, this was NOT the case. Here is the scenario:

  1. We had created a schedule back in July/August of last year (2024) and configured it to run one of our workflows daily at 5am UTC. The version of temporal server we were running at that time was either 1.20.3 or 1.21.4 (self hosted).
  2. We had edited/updated the schedule a couple of times using the Temporal UI shortly after it was created and never ran into any issues.
  3. Last Friday, 4/11/25, we updated the schedule using the Temporal UI, changing the value of one of the input attributes (not a new attribute).
  4. Upon checking the scheduled workflow executions on Monday morning (4/14/25), we saw that the three (3) scheduled workflow runs for Saturday/Sunday/Monday had errored.
  5. The error displayed was unable to decode: json: cannot unmarshal object into Go value of type string. Manually triggering the workflow run from the schedule would also result in the same error. Looking at the event history for the scheduled workflow run, five (5) entries exist - which have been included in the “Redacted Event History” below.
  6. Our first assumption was that we must have accidentally changed an input attribute name or data type. We did a thorough analysis and firmly concluded this was not the case. The input was valid. This was further confirmed by:
    1. Manually starting the workflow from the client api using the same exact input as the schedule. The workflow started and ran without any errors.
    2. Creating a new exact copy of the schedule with the same input. The workflow started and ran without any errors.
  7. Based on bullet #6 directly above, we decided to test updating one of our other schedules created around the same time last year. From the Temporal UI, we simply clicked the “Edit” option from the menu in the top right corner of the schedule page. Then, without actually changing anything, clicked the “Update” button. We then “triggered” an immediate run of the workflow from the schedule. :collision: - we immediately saw the same error.

So, it appears to us that there is a bug somewhere in the temporal scheduler workflow. Maybe related to old/existing schedules and updating those schedules when on a newer version of Temporal server? That is our theory anyway… We are currently running Temporal server version 1.27.2.

One theory we have as to why that mentioned error is occurring may be due to the schedulespb.StartScheduleArgs that are used by the SchedulerWorkflow. There was a recent Pull Request (#7528) that changed the attributes on that struct.

We’ve worked around the issue by recreating our impacted schedules. However, we are hoping someone with a deeper understanding of temporal schedules could point us to the underlying root cause of this error. Thanks!

Redacted Event History 
[
  {
    "eventId": "1",
    "eventTime": "2025-04-14T05:00:00.131286881Z",
    "eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_STARTED",
    "taskId": "192947305",
    "workflowExecutionStartedEventAttributes": {
      "workflowType": {"name": "WORKFLOW_NAME"},
      "taskQueue": {
        "name": "WORKFLOW_NAME",
        "kind": "TASK_QUEUE_KIND_NORMAL"
      },
      "input": {
        "payloads": [
          {
            "metadata": {"encoding": "anNvbi9wbGFpbg=="},
            "data": "REDACTED - verified to be valid when base64 decoded"
          }
        ]
      },
      "workflowExecutionTimeout": "0s",
      "workflowRunTimeout": "0s",
      "workflowTaskTimeout": "10s",
      "continuedFailure": {
        "message": "unable to decode: json: cannot unmarshal object into Go value of type string",
        "source": "GoSDK",
        "cause": {
          "message": "unable to decode",
          "source": "GoSDK",
          "applicationFailureInfo": {}
        },
        "applicationFailureInfo": {"type": "wrapError"}
      },
      "lastCompletionResult": {
        "payloads": [
          {
            "metadata": {"encoding": "anNvbi9wbGFpbg=="},
            "data": "REDACTED - verified to be valid when base64 decoded"
          }
        ]
      },
      "originalExecutionRunId": "019632ab-2903-745d-8d41-00079674dc1b",
      "identity": "temporal-scheduler-namespace-schedule_name",
      "firstExecutionRunId": "019632ab-2903-745d-8d41-00079674dc1b",
      "attempt": 1,
      "firstWorkflowTaskBackoff": "0s",
      "searchAttributes": {
        "indexedFields": {
          "TemporalScheduledById": {
            "metadata": {"encoding": "anNvbi9wbGFpbg==", "type": "S2V5d29yZA=="},
            "data": "InBwbXMtcGlwZWxpbmUi"
          },
          "TemporalScheduledStartTime": {
            "metadata": {"encoding": "anNvbi9wbGFpbg==", "type": "RGF0ZXRpbWU="},
            "data": "IjIwMjUtMDQtMTRUMDU6MDA6MDBaIg=="
          }
        }
      },
      "header": {
        "fields": {
          "_tracer-data": {
            "metadata": {"encoding": "anNvbi9wbGFpbg=="},
            "data": "eyJtZXRhZGF0YSI6eyJlbmNvZGluZyI6ImFuTnZiaTl3YkdGcGJnPT0ifSwiZGF0YSI6ImV5SjBjbUZqWlhCaGNtVnVkQ0k2SWpBd0xXUmhZekkwTkRBek9UQTJaR05qTlRNeE5XRTVZMk01TTJZM1pHRm1Oamt5TFdabU1ERTRNalk1TVdSaU9UVTVORGt0TURFaWZRPT0ifQ=="
          },
          "tempulse.Fields": {
            "metadata": {"encoding": "anNvbi9wbGFpbg=="},
            "data": "eyJtZXRhZGF0YSI6eyJlbmNvZGluZyI6ImFuTnZiaTl3YkdGcGJnPT0ifSwiZGF0YSI6ImV5SnpjR0Z1WDJsa0lqb2labVl3TVRneU5qa3haR0k1TlRrME9TSXNJblJ5WVdObFgybGtJam9pWkdGak1qUTBNRE01TURaa1kyTTFNekUxWVRsall6a3paamRrWVdZMk9USWlmUT09In0="
          }
        }
      },
      "workflowId": "WORKFLOW_ID"
    }
  },
  {
    "eventId": "2",
    "eventTime": "2025-04-14T05:00:00.131378976Z",
    "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
    "taskId": "192947306",
    "workflowTaskScheduledEventAttributes": {
      "taskQueue": {"name": "WORKFLOW_NAME", "kind": "TASK_QUEUE_KIND_NORMAL"},
      "startToCloseTimeout": "10s",
      "attempt": 1
    }
  },
  {
    "eventId": "3",
    "eventTime": "2025-04-14T05:00:00.159842277Z",
    "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
    "taskId": "192947311",
    "workflowTaskStartedEventAttributes": {
      "scheduledEventId": "2",
      "identity": "REDACTED",
      "requestId": "b9142f28-ae33-4851-98f5-7b0e4fe73ce6",
      "historySizeBytes": "2844",
      "workerVersion": {"buildId": "cb05e9920f7c564ce66c300ae1e122fe"}
    }
  },
  {
    "eventId": "4",
    "eventTime": "2025-04-14T05:00:00.187024788Z",
    "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
    "taskId": "192947315",
    "workflowTaskCompletedEventAttributes": {
      "scheduledEventId": "2",
      "startedEventId": "3",
      "identity": "REDACTED",
      "workerVersion": {"buildId": "cb05e9920f7c564ce66c300ae1e122fe"},
      "sdkMetadata": {
        "langUsedFlags": [3],
        "sdkName": "temporal-go",
        "sdkVersion": "1.33.1"
      },
      "meteringMetadata": {}
    }
  },
  {
    "eventId": "5",
    "eventTime": "2025-04-14T05:00:00.187102747Z",
    "eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_FAILED",
    "taskId": "192947316",
    "workflowExecutionFailedEventAttributes": {
      "failure": {
        "message": "unable to decode: json: cannot unmarshal object into Go value of type string",
        "source": "GoSDK",
        "cause": {
          "message": "unable to decode",
          "source": "GoSDK",
          "applicationFailureInfo": {}
        },
        "applicationFailureInfo": {"type": "wrapError"}
      },
      "retryState": "RETRY_STATE_RETRY_POLICY_NOT_SET",
      "workflowTaskCompletedEventId": "4"
    }
  }
]