CRON schedule issue in higher environment

Hi Maxim/Thiomir

I am facing an issue in SIT environment where I have a CRON job that runs in every 1 hour, it works fine for that day but from next day it gets stucked.

Please help to resolve this.

Temporal version : 2.10.3

Thanks.

but from next day it gets stucked

Can you give more info please? If you can share event history of this “stuck” execution would help.

Hi @tihomir

Please find the requested details.


{
  "events": [
    {
      "eventId": "1",
      "eventTime": "2023-04-08T00:41:17.768625578Z",
      "eventType": "WorkflowExecutionStarted",
      "version": "0",
      "taskId": "22790799383",
      "workerMayIgnore": false,
      "workflowExecutionStartedEventAttributes": {
        "workflowType": {
          "name": "ProcessFileWorkFlow"
        },
        "parentWorkflowNamespace": "",
        "parentWorkflowNamespaceId": "",
        "parentWorkflowExecution": null,
        "parentInitiatedEventId": "0",
        "taskQueue": {
          "name": "rjil.process.autorefill.readfile",
          "kind": "Normal"
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "Ii9USUJDTy9QQVlNRU5UL09VVEJPVU5EL1RFTVAvIg=="
            }
          ]
        },
        "workflowExecutionTimeout": "0s",
        "workflowRunTimeout": "0s",
        "workflowTaskTimeout": "10s",
        "continuedExecutionRunId": "e24ff264-558e-4e14-95b6-b4755839976e",
        "initiator": "CronSchedule",
        "continuedFailure": null,
        "lastCompletionResult": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": "e30="
            }
          ]
        },
        "originalExecutionRunId": "232af0ae-8c2f-47c9-9164-3e3015db462d",
        "identity": "",
        "firstExecutionRunId": "579dced8-94d0-4c69-b2d6-3a6e3fc839e1",
        "retryPolicy": null,
        "attempt": 1,
        "workflowExecutionExpirationTime": null,
        "cronSchedule": "@every 1h",
        "firstWorkflowTaskBackoff": "20.714638187s",
        "memo": null,
        "searchAttributes": null,
        "prevAutoResetPoints": {
          "points": []
        },
        "header": {
          "fields": {}
        },
        "parentInitiatedEventVersion": "0"
      }
    }
  ]
}

Hi @tihomir

Any updates.

Might be potential timer loss issue as first workflow task should have been scheduled according to your cron schedule definition.

From client can you try to SignalWithStart this execution (can send a dummy signal) and see if this unblocks it?

Hi @tihomir

Did not get the point how will I know that my workflow execution is not completed or stuck because it does not give me any error/exception on cluster console. It’s behavior is like for the whole it will work fine but as it goes to next day it will 2-3 times then will get stucked without any exception and error. The same code is working fine in my development environment, there it did not give me any issue.

Hi @tihomir /Maxim

Any updates because we are not able to find the issue.

Thanks.