Implementing a Wait Until

Community Support
1

I’m trying to implement a wait until a specified time feature.
In case of failure in the worker process, replaying adds a 5 second delay. Why?

Suppose current time is 14:00:00.
I’m waiting 10 seconds, and then I’m waiting until 14:00:50. Then printing something.
At time 14:00:15, I stopped the worker.
At time 14:00:35, I ran the worker.

At 14:00:55, the worker prints something.

I have tested it twice.
In fact, I expect 14:00:50.
Or 14:01:10 (a diff time for the application restarting approx. 20 sec.)

My code:

private void sleep(WaitState state, JsonUtil.JsonDataHolder input) {
        WaitFor waitFor = state.getWaitFor();
        long timeToWait = 0;
        if (waitFor instanceof WaitForSeconds waitForSeconds) {
            timeToWait = waitForSeconds.getSeconds() * 1000;
        }   
        else if (waitFor instanceof WaitForTimestamp waitForTimestamp) {
            Instant waitUntil = waitForTimestamp.getTimestamp().toInstant();
            timeToWait = Instant.ofEpochMilli(Workflow.currentTimeMillis()).until(waitUntil, ChronoUnit.MILLIS);
        } 

        log.trace("Sleep {} ms.", timeToWait);

        if (timeToWait > 0) {
            Workflow.sleep(timeToWait);
        }
}
2

Would you post the workflow execution history of the workflow?

3

I started the workflow at 49:00
I stopped the worker at approx. 49:30
My aim is to print something at approx. 49:50

Maybe it is related with step 13.
WorkflowTaskTimedOut 1m 5s (+5s)

image
image1206×977 154 KB

Because of the limitation I will post the remaining in the next post:

{ "events": [{"eventId":"1","eventTime":"2022-02-27T19:49:00.000Z","eventType":"WorkflowExecutionStarted","version":"0","taskId":"2104947","workflowExecutionStartedEventAttributes":{"workflowType":{"name":"test"},"parentWorkflowNamespace":"","parentInitiatedEventId":"0","taskQueue":{"name":"code2-flow","kind":"Normal"},"input":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"InRlc3Qi"},{"metadata":{"encoding":"YmluYXJ5L251bGw="},"data":""},{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"IntcIkNvbW1lbnRcIjpcIkEgZGVzY3JpcHRpb24gb2YgbXkgc3RhdGUgbWFjaGluZVwiLFwiU3RhcnRBdFwiOlwiV2FpdFwiLFwiVGltZW91dFNlY29uZHNcIjoxODAsXCJTdGF0ZXNcIjp7XCJXYWl0XCI6e1wiU2Vjb25kc1wiOjEwLFwiSW5wdXRQYXRoXCI6XCIkLmZpcnN0XCIsXCJOZXh0XCI6XCJXYWl0IFVudGlsXCIsXCJUeXBlXCI6XCJXYWl0XCJ9LFwiV2FpdCBVbnRpbFwiOntcIlRpbWVzdGFtcFBhdGhcIjpcIiQud2FpdFVudGlsXCIsXCJOZXh0XCI6XCJQcmludFwiLFwiVHlwZVwiOlwiV2FpdFwifSxcIlByaW50XCI6e1wiRW5kXCI6dHJ1ZSxcIlR5cGVcIjpcIlBhc3NcIn19fSI="},{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"IntcbiAgXCJDb21tZW50XCI6IFwiSW5zZXJ0IHlvdXIgSlNPTiBoZXJlXCIsXG4gIFwiZmlyc3RcIjoge1xuICAgIFwid2FpdFwiOiAxMixcbiAgICBcIndhaXRVbnRpbFwiOiBcIjIwMjItMDItMjdUMjI6NTA6MDArMDM6MDBcIixcbiAgICBcImFcIjogXCJ2YXJpYWJsZVwiLFxuICAgIFwiYlwiOiBbXG4gICAgICAyLFxuICAgICAgMyxcbiAgICAgIDRcbiAgICBdLFxuICAgIFwiY1wiOiB7XG4gICAgICBcInhcIjogMSxcbiAgICAgIFwieVwiOiAxXG4gICAgfVxuICB9XG59Ig=="}]},"workflowExecutionTimeout":"180s","workflowRunTimeout":"180s","workflowTaskTimeout":"10s","continuedExecutionRunId":"","initiator":"Unspecified","originalExecutionRunId":"51f15e42-01cd-4c7f-bd92-ded1daf7a3ff","identity":"15298@MacBook-Pro-4.home","firstExecutionRunId":"51f15e42-01cd-4c7f-bd92-ded1daf7a3ff","retryPolicy":{"nonRetryableErrorTypes":[],"initialInterval":"1s","backoffCoefficient":2,"maximumInterval":"100s","maximumAttempts":3},"attempt":1,"workflowExecutionExpirationTime":"2022-02-27T19:52:00.000Z","cronSchedule":"","firstWorkflowTaskBackoff":"0s","header":{"fields":{}}}},{"eventId":"2","eventTime":"2022-02-27T19:49:00.000Z","eventType":"WorkflowTaskScheduled","version":"0","taskId":"2104948","workflowTaskScheduledEventAttributes":{"taskQueue":{"name":"code2-flow","kind":"Normal"},"startToCloseTimeout":"10s","attempt":1}},{"eventId":"3","eventTime":"2022-02-27T19:49:00.000Z","eventType":"WorkflowTaskStarted","version":"0","taskId":"2104953","workflowTaskStartedEventAttributes":{"scheduledEventId":"2","identity":"15284@MacBook-Pro-4.home","requestId":"195a941a-8952-4993-aef7-ceb0483ace13"}},{"eventId":"4","eventTime":"2022-02-27T19:49:01.000Z","eventType":"WorkflowTaskCompleted","version":"0","taskId":"2104956","workflowTaskCompletedEventAttributes":{"scheduledEventId":"2","startedEventId":"3","identity":"15284@MacBook-Pro-4.home","binaryChecksum":""}},{"eventId":"5","eventTime":"2022-02-27T19:49:01.000Z","eventType":"TimerStarted","version":"0","taskId":"2104957","timerStartedEventAttributes":{"timerId":"27ef97a5-72e9-3077-84c0-66c8e7a3c444","startToFireTimeout":"10s","workflowTaskCompletedEventId":"4"}},{"eventId":"6","eventTime":"2022-02-27T19:49:11.000Z","eventType":"TimerFired","version":"0","taskId":"2104960","timerFiredEventAttributes":{"timerId":"27ef97a5-72e9-3077-84c0-66c8e7a3c444","startedEventId":"5"}},{"eventId":"7","eventTime":"2022-02-27T19:49:11.000Z","eventType":"WorkflowTaskScheduled","version":"0","taskId":"2104961","workflowTaskScheduledEventAttributes":{"taskQueue":{"name":"15284@MacBook-Pro-4.home:efd9ab2e-bb5a-4bca-a4cc-09923bbfec63","kind":"Sticky"},"startToCloseTimeout":"10s","attempt":1}},{"eventId":"8","eventTime":"2022-02-27T19:49:11.000Z","eventType":"WorkflowTaskStarted","version":"0","taskId":"2104965","workflowTaskStartedEventAttributes":{"scheduledEventId":"7","identity":"efd9ab2e-bb5a-4bca-a4cc-09923bbfec63","requestId":"cefe3726-8760-44e9-a444-e6a1fe426dd1"}},{"eventId":"9","eventTime":"2022-02-27T19:49:11.000Z","eventType":"WorkflowTaskCompleted","version":"0","taskId":"2104968","workflowTaskCompletedEventAttributes":{"scheduledEventId":"7","startedEventId":"8","identity":"15284@MacBook-Pro-4.home","binaryChecksum":""}},{"eventId":"10","eventTime":"2022-02-27T19:49:11.000Z","eventType":"TimerStarted","version":"0","taskId":"2104969","timerStartedEventAttributes":{"timerId":"346dc847-9f2b-3eaa-9a16-135eebcd8d88","startToFireTimeout":"48s","workflowTaskCompletedEventId":"9"}},{"eventId":"11","eventTime":"2022-02-27T19:50:00.000Z","eventType":"TimerFired","version":"0","taskId":"2104972","timerFiredEventAttributes":{"timerId":"346dc847-9f2b-3eaa-9a16-135eebcd8d88","startedEventId":"10"}},{"eventId":"12","eventTime":"2022-02-27T19:50:00.000Z","eventType":"WorkflowTaskScheduled","version":"0","taskId":"2104973","workflowTaskScheduledEventAttributes":{"taskQueue":{"name":"15284@MacBook-Pro-4.home:efd9ab2e-bb5a-4bca-a4cc-09923bbfec63","kind":"Sticky"},"startToCloseTimeout":"10s","attempt":1}},{"eventId":"13","eventTime":"2022-02-27T19:50:05.000Z","eventType":"WorkflowTaskTimedOut","version":"0","taskId":"2104977","workflowTaskTimedOutEventAttributes":{"scheduledEventId":"12","startedEventId":"0","timeoutType":"ScheduleToStart"}},{"eventId":"14","eventTime":"2022-02-27T19:50:05.000Z","eventType":"WorkflowTaskScheduled","version":"0","taskId":"2104978","workflowTaskScheduledEventAttributes":{"taskQueue":{"name":"code2-flow","kind":"Normal"},"startToCloseTimeout":"10s","attempt":1}},{"eventId":"15","eventTime":"2022-02-27T19:50:05.000Z","eventType":"WorkflowTaskStarted","version":"0","taskId":"2104981","workflowTaskStartedEventAttributes":{"scheduledEventId":"14","identity":"15323@MacBook-Pro-4.home","requestId":"a6f2117c-2a92-4a58-ad1d-3c4a911902cb"}},{"eventId":"16","eventTime":"2022-02-27T19:50:05.000Z","eventType":"WorkflowTaskCompleted","version":"0","taskId":"2104984","workflowTaskCompletedEventAttributes":{"scheduledEventId":"14","startedEventId":"15","identity":"15323@MacBook-Pro-4.home","binaryChecksum":""}},{"eventId":"17","eventTime":"2022-02-27T19:50:05.000Z","eventType":"WorkflowExecutionCompleted","version":"0","taskId":"2104985","workflowExecutionCompletedEventAttributes":{"result":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"IntcIndhaXRcIjoxMixcIndhaXRVbnRpbFwiOlwiMjAyMi0wMi0yN1QyMjo1MDowMCswMzowMFwiLFwiYVwiOlwidmFyaWFibGVcIixcImJcIjpbMiwzLDRdLFwiY1wiOntcInhcIjoxLFwieVwiOjF9fSI="}]},"workflowTaskCompletedEventId":"16","newExecutionRunId":""}}] }
4

Step 8-17

image
image1434×976 159 KB

5

This is due to workflow caching which we call sticky execution.

This post has more details.

6

Thanks for your reply, maxim. I will read the sticky execution articles.

If I were use System.currentTimeInMillis and use java scheduledexecutorservice, I think it could work. But these should be discouraged methods in workflows.

Is there a way to implement wait until feature in a sub second resolution?

7

If you need a hard guarantee that a workflow recovers from a worker crash under one second then Temporal is not the right technology. You can change the sticky queue schedule to start timeout to a lower number, but having it under one second is going to lead to too many spurious timeouts which can negatively affect the latency of workflows and overall system performance.

1 Like