Scheduled Workflow Instance Delayed by One Minute - Seeking Insights

Hi Temporal Community,

I’m currently facing an issue with a scheduled instance in Temporal, and I’m hoping to get some insights or advice on what might be causing this.

Details:

  • Temporal SDK Version: 1.20
  • SDK: Java SDK
  • Scheduled Time: The schedule instance had an action scheduled to kick off a workflow at 14:30:00.882Z (with jitter).
  • Actual Start Time: The action did not start until 14:31:00.238Z, which is when the workflow kicked off.

Question: Why did the action not kick off until 14:31? What could be causing this one-minute delay?

Additional Context:

  • System Load: At the time of scheduling, the system load appeared normal, and there were no resource constraints.
  • Time Synchronization: All servers are synchronized so clock drift should not be an issue.
  • Configuration: The polling interval for worker nodes is set to a reasonable value, and there have been no recent changes to the Temporal configuration.

I’ve checked the Temporal server logs and didn’t find any errors or warnings around the time the workflow was supposed to be scheduled. The logs show the workflow was indeed scheduled at 14:30:00.882Z but only started at 14:31:00.238Z.

Has anyone else encountered similar issues or have any ideas on what might be causing this delay? Any advice or troubleshooting tips would be greatly appreciated.

Thank you!