Issue with Temporal Scheduler Skipping Executions

Community Support
, , , ,
1

I’m experiencing an issue with Temporal Schedules and would appreciate your insights. Here’s the scenario:

  • I have a scheduled Temporal workflow that triggers every Saturday at 1:00 AM EST.
  • March 8th, 2025, was the run from previous week. It triggered fine at the scheduled time as expected.
    • There was an application issue causing the workflow activity to fail after retries.
    • I had to manually rerun the failed workflow from the UI, and it went fine at 9:00 AM EST on the same day.
  • The next scheduled workflow which was supposed to run on March 15th, 2025 did not execute at all. I can see the workflow run in the upcoming runs and it never triggered even though its time has passed.
  • Some of the other schedules using the same task queue / namespace / workers ran fine.
  • Can the manual triggers of a failed workflow cause the next schedule not to run? I tried to replicate this with a test workflow and couldn’t replicate it. It seems to randomly not trigger a specific schedule. Any leads to find details on why it didn’t trigger will help as it is affecting our production schedules.

As you can see below, today is 16th March 2025 and I still see it in upcoming runs. Time zone is America/New York. Postrges is used as a backend and this is a self hosted Temporal service.

Screenshot 2025-03-16 at 11.15.53 PM
Screenshot 2025-03-16 at 11.15.53 PM3096×672 194 KB

Versions:
Temporal UI 2.31.2
Temporal Server Version 1.25.1
Springboot Temporal Alpha 1.23.2

Schedule Spec:
{“second”:[{“step”:1}],“minute”:[{“step”:1}],“hour”:[{“step”:1}],“dayOfMonth”:[{“start”:1,“end”:31,“step”:1}],“month”:[{“start”:1,“end”:12,“step”:1}],“dayOfWeek”:[{“start”:6,“end”:6,“step”:1}]}

Advanced Settings:
Schedule Start Time: None
Schedule End Time: 2026-01-31T00:00:00Z
Jitter: None
Exclusion Calendar: None
Overlap Policy: SCHEDULE_OVERLAP_POLICY_SKIP

2

Was able to resolve this by

  1. Increasing catchup window ( to handle any activity failures)
  2. Increased workflowRunTimeOut
  3. Increased Max Attempts
    The existing workflows had to be updated to change the specs/policy using the schedule update APIs