Temporal Workflow Error: Unable to Decode Input Payload for Workflow After New Version Deployment

Community Support
,
1

Hey Temporal Community, :wave:

We recently deployed a new version of our Temporal workflows, specifically OrderWorkflowV2 and its child workflow PaymentWorkflowV2. After the deployment, we observed an issue where previously running workflows are failing to resume, while newly initiated workflows are working fine.

The error we are encountering is:

unable to decode the workflow function input payload with error: payload item 0: unable to decode: parsing time "" as "2006-01-02T15:04:05Z07:00": cannot parse "" as "2006", function name: PaymentWorkflowV2.

What We Know So Far:

  1. This error only occurs for workflows that were initiated before the deployment.
  2. Newly initiated workflows after the deployment are working fine.
  3. The error seems to be related to Temporal trying to parse an empty string as a timestamp using the format 2006-01-02T15:04:05Z07:00.
  4. We did not change the PaymentWorkflowInput structure or the workflow signature between the old and new versions.

Workflow Changes We Made:

We added the following block of code to our PaymentWorkflowV2 workflow to handle versioning logic using GetVersion:

ver := workflow.GetVersion(ctx, "fetchTxnData_V1", workflow.DefaultVersion, 0)
if ver != workflow.DefaultVersion {
    txnInfo, err := utils.RetrieveTxnInfo(ctx, response.OrderInfo.TxnID, services.PaymentWorkflow.Activities)
    if err != nil {
        return err
    }
    if len(txnInfo.History) > 0 && txnInfo.History[0].State == entity.TxnCompleted {
        basket, err := utils.FetchBasketInfo(ctx, response.OrderInfo.BasketID, services.PaymentWorkflow.Activities)
        if err != nil {
            return err
        }
        input.BasketData = *basket.Data
        notifyErr := utils.SendAlert(ctx, input, services.PaymentWorkflow.Activities)
        if notifyErr != nil {
            return notifyErr
        }

        services.log.Infof("OrderWorkflowV2 | Workflow data --------  :%+v", input.HideSensitiveData())
        return nil
    }
}

We used Temporal’s versioning API to ensure backward compatibility with older workflows.

Our Workflow Input Structure:

Here’s the input structure we’re passing to the PaymentWorkflowV2 workflow:

type WorkflowArgs struct {
    TransactionKey       string
    BasketKey            string
    ClientState          string
    OrderReference       string
    PaymentMetadata      PaymentDetails
    BasketData           BasketInfo
    ValidUntil           time.Time
    OrderCreatedOn       time.Time
    PaymentState         string
    PaymentProvider      enums.ProviderType
    ClientID             string
    WorkflowSettings     TaskConfig
}

Here’s the PaymentDetails structure:

type PaymentDetails struct {
    BankTxnID       string
    Status          string
    PaymentDate     string
    VendorState     string
    Method          string
    MethodType      string
    GatewayTxnID    string
    VendorTxnID     string
    Provider        string
    ProviderID      string
    VendorResponse  datatypes.JSONType[VendorResponseData]
    WebhookResponse datatypes.JSONType[WebhookPayload]
}

The Problem:

We did not make any changes to the PaymentWorkflowInput structure, so we are unsure why this parsing error is happening. The field causing the error is likely one of the time fields (ValidUntil or OrderCreatedOn) in the WorkflowArgs struct.

However:

  • We confirmed that new workflows are working fine with the same input structure.
  • The issue only occurs when previously running workflows try to resume after the deployment.

Questions for the Community:

  1. Could this issue be related to how Temporal persists and deserializes workflow state when resuming old workflows after a deployment?
  2. Is there a way to gracefully handle empty or missing time fields during workflow resumption?
  3. Are there any best practices for using Temporal’s versioning APIs to avoid such issues during deployments?

We’d greatly appreciate any guidance or suggestions! :pray:

Thanks in advance!
Ridam Garg