Hi,
I am able to successfully start a workflow with cron enabled. But after rebooting the service/microservices, the activity corresponding to the workflow is not being invoked though the status of workflow is WORKFLOW_EXECUTION_STATUS_RUNNING. Is this a known issue? If so, is there any workaround to handle this?
Could you share the event history of this execution please?
[{"eventId":"1","eventTime":{"seconds":"1678488239","nanos":57327435},"eventType":"EVENT_TYPE_WORKFLOW_EXECUTION_STARTED","version":"0","taskId":"6304120","workflowExecutionStartedEventAttributes":{"workflowType":{"name":"IntegrationWorkflow"},"parentWorkflowNamespace":"","parentInitiatedEventId":"0","taskQueue":{"name":"IntegrationWorkflowTaskQueue","kind":"TASK_QUEUE_KIND_NORMAL"},"input":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"MzE="},{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"ImNjOWJjYThlLTI2NGYtNGQ3NS05ZTk1LThmYTc2YTk3YmFlMCI="}]},"workflowExecutionTimeout":{"seconds":"0","nanos":0},"workflowRunTimeout":{"seconds":"300","nanos":0},"workflowTaskTimeout":{"seconds":"10","nanos":0},"continuedExecutionRunId":"55424c74-fdfd-480a-b5da-542920e5a419","initiator":"CONTINUE_AS_NEW_INITIATOR_CRON_SCHEDULE","originalExecutionRunId":"66122026-eeea-4612-9b57-288672f6b6a8","identity":"","firstExecutionRunId":"0304e4ca-5cee-4046-8445-02dd7d5e654e","attempt":1,"cronSchedule":"@every 60s","firstWorkflowTaskBackoff":{"seconds":"59","nanos":0},"prevAutoResetPoints":{"points":[]},"header":{"fields":{}}},"attributes":"workflowExecutionStartedEventAttributes"},{"eventId":"2","eventTime":{"seconds":"1678488298","nanos":60821162},"eventType":"EVENT_TYPE_WORKFLOW_TASK_SCHEDULED","version":"0","taskId":"6304128","workflowTaskScheduledEventAttributes":{"taskQueue":{"name":"IntegrationWorkflowTaskQueue","kind":"TASK_QUEUE_KIND_NORMAL"},"startToCloseTimeout":{"seconds":"10","nanos":0},"attempt":1},"attributes":"workflowTaskScheduledEventAttributes"},{"eventId":"3","eventTime":{"seconds":"1678488298","nanos":82119380},"eventType":"EVENT_TYPE_WORKFLOW_TASK_STARTED","version":"0","taskId":"6304131","workflowTaskStartedEventAttributes":{"scheduledEventId":"2","identity":"1@tenant-manager-5b9b85dcf8-pw9wc","requestId":"54b0d53c-507a-4a14-b03e-54123ca6a581"},"attributes":"workflowTaskStartedEventAttributes"},{"eventId":"4","eventTime":{"seconds":"1678488298","nanos":109008857},"eventType":"EVENT_TYPE_WORKFLOW_TASK_COMPLETED","version":"0","taskId":"6304134","workflowTaskCompletedEventAttributes":{"scheduledEventId":"2","startedEventId":"3","identity":"1@tenant-manager-5b9b85dcf8-pw9wc","binaryChecksum":""},"attributes":"workflowTaskCompletedEventAttributes"},{"eventId":"5","eventTime":{"seconds":"1678488298","nanos":109059708},"eventType":"EVENT_TYPE_ACTIVITY_TASK_SCHEDULED","version":"0","taskId":"6304135","activityTaskScheduledEventAttributes":{"activityId":"45353ecd-c8d9-3a47-a843-c5808d074d7e","activityType":{"name":"GenerateIntegrationEvents"},"namespace":"","taskQueue":{"name":"IntegrationWorkflowTaskQueue","kind":"TASK_QUEUE_KIND_NORMAL"},"header":{"fields":{}},"input":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"MzE="},{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"ImNjOWJjYThlLTI2NGYtNGQ3NS05ZTk1LThmYTc2YTk3YmFlMCI="}]},"scheduleToCloseTimeout":{"seconds":"300","nanos":0},"scheduleToStartTimeout":{"seconds":"300","nanos":0},"startToCloseTimeout":{"seconds":"300","nanos":0},"heartbeatTimeout":{"seconds":"0","nanos":0},"workflowTaskCompletedEventId":"4","retryPolicy":{"nonRetryableErrorTypes":[],"initialInterval":{"seconds":"1","nanos":0},"backoffCoefficient":2,"maximumInterval":{"seconds":"100","nanos":0},"maximumAttempts":0}},"attributes":"activityTaskScheduledEventAttributes"},{"eventId":"6","eventTime":{"seconds":"1678488298","nanos":133740242},"eventType":"EVENT_TYPE_ACTIVITY_TASK_STARTED","version":"0","taskId":"6304138","activityTaskStartedEventAttributes":{"scheduledEventId":"5","identity":"1@tenant-manager-5b9b85dcf8-pw9wc","requestId":"37fb298b-efea-425b-a101-83e006355886","attempt":1},"attributes":"activityTaskStartedEventAttributes"},{"eventId":"7","eventTime":{"seconds":"1678488298","nanos":236271328},"eventType":"EVENT_TYPE_ACTIVITY_TASK_COMPLETED","version":"0","taskId":"6304139","activityTaskCompletedEventAttributes":{"result":{"payloads":[{"metadata":{"encoding":"YmluYXJ5L251bGw="},"data":""}]},"scheduledEventId":"5","startedEventId":"6","identity":"1@tenant-manager-5b9b85dcf8-pw9wc"},"attributes":"activityTaskCompletedEventAttributes"},{"eventId":"8","eventTime":{"seconds":"1678488298","nanos":236291088},"eventType":"EVENT_TYPE_WORKFLOW_TASK_SCHEDULED","version":"0","taskId":"6304140","workflowTaskScheduledEventAttributes":{"taskQueue":{"name":"1@tenant-manager-5b9b85dcf8-pw9wc:16e8a465-1f54-4013-90fe-2679fc60d9b6","kind":"TASK_QUEUE_KIND_STICKY"},"startToCloseTimeout":{"seconds":"10","nanos":0},"attempt":1},"attributes":"workflowTaskScheduledEventAttributes"},{"eventId":"9","eventTime":{"seconds":"1678488298","nanos":259605373},"eventType":"EVENT_TYPE_WORKFLOW_TASK_STARTED","version":"0","taskId":"6304144","workflowTaskStartedEventAttributes":{"scheduledEventId":"8","identity":"16e8a465-1f54-4013-90fe-2679fc60d9b6","requestId":"200adc20-772a-46cd-8baa-5d0991ddf219"},"attributes":"workflowTaskStartedEventAttributes"},{"eventId":"10","eventTime":{"seconds":"1678488298","nanos":283832464},"eventType":"EVENT_TYPE_WORKFLOW_TASK_COMPLETED","version":"0","taskId":"6304147","workflowTaskCompletedEventAttributes":{"scheduledEventId":"8","startedEventId":"9","identity":"1@tenant-manager-5b9b85dcf8-pw9wc","binaryChecksum":""},"attributes":"workflowTaskCompletedEventAttributes"},{"eventId":"11","eventTime":{"seconds":"1678488298","nanos":283916807},"eventType":"EVENT_TYPE_WORKFLOW_EXECUTION_CONTINUED_AS_NEW","version":"0","taskId":"6304148","workflowExecutionContinuedAsNewEventAttributes":{"newExecutionRunId":"2e9e26f0-38db-4d92-917e-889dd0ee02e1","workflowType":{"name":"IntegrationWorkflow"},"taskQueue":{"name":"IntegrationWorkflowTaskQueue","kind":"TASK_QUEUE_KIND_NORMAL"},"input":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"MzE="},{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"ImNjOWJjYThlLTI2NGYtNGQ3NS05ZTk1LThmYTc2YTk3YmFlMCI="}]},"workflowRunTimeout":{"seconds":"300","nanos":0},"workflowTaskTimeout":{"seconds":"10","nanos":0},"workflowTaskCompletedEventId":"10","backoffStartInterval":{"seconds":"60","nanos":0},"initiator":"CONTINUE_AS_NEW_INITIATOR_CRON_SCHEDULE","header":{"fields":{}}},"attributes":"workflowExecutionContinuedAsNewEventAttributes"}]
@tihomir Does this help?
Thanks yes, I’d like to see tho not this one but the one thats still currently in Running status (one that was created right after the one you shared completed i guess).The history you shared is for a completed execution (continued as new)
[{"eventId":"1","eventTime":{"seconds":"1678492991","nanos":735560747},"eventType":"EVENT_TYPE_WORKFLOW_EXECUTION_STARTED","version":"0","taskId":"6305644","workflowExecutionStartedEventAttributes":{"workflowType":{"name":"IntegrationWorkflow"},"parentWorkflowNamespace":"","parentInitiatedEventId":"0","taskQueue":{"name":"IntegrationWorkflowTaskQueue","kind":"TASK_QUEUE_KIND_NORMAL"},"input":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"MzE="},{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"ImE2MTE5ZDYxLTQ2Y2EtNDc5OS1hOWYyLWFmN2ZjNGZlMzY0ZCI="}]},"workflowExecutionTimeout":{"seconds":"0","nanos":0},"workflowRunTimeout":{"seconds":"300","nanos":0},"workflowTaskTimeout":{"seconds":"10","nanos":0},"continuedExecutionRunId":"f1c0c8e7-528d-44b2-8133-5b3d4af94457","initiator":"CONTINUE_AS_NEW_INITIATOR_CRON_SCHEDULE","continuedFailure":{"message":"workflow timeout","source":"Server","stackTrace":"","timeoutFailureInfo":{"timeoutType":"TIMEOUT_TYPE_START_TO_CLOSE"},"failureInfo":"timeoutFailureInfo"},"originalExecutionRunId":"e8673a2c-9736-4817-8805-30ca4740d93f","identity":"","firstExecutionRunId":"ca28ec90-729e-4a8f-b37c-08b448b0a2d2","attempt":1,"cronSchedule":"@every 60s","firstWorkflowTaskBackoff":{"seconds":"60","nanos":0},"prevAutoResetPoints":{"points":[]},"header":{"fields":{}}},"attributes":"workflowExecutionStartedEventAttributes"}]
After creating, the above workflow was running as expected. But after the application restart, I don’t see the corresponding activity being invoked.
Ok this execution has a 60s firstWorkflowTaskBackoff, meaning that after 60s of when it was created (you can see the timestamp property of the first WorkflowExecutionStarted event to see when the execution was created. 60s after service should have recorded the first WorkflowTaskScheduled event in history.
Can you try to “kick” it with SignalWithStart see if this helps? So for example from client:
WorkflowStub stub = client.newUntypedWorkflowStub("<workflow_id>");
stub.signalWithStart("someSignalName", new Object[]{}, new Object[]{});
replace <workflow_id> with workflow id of this last execution you shared. It does not matter what signal name you set with signalWithStart when trying this (you dont have to have a signal handler for the signal name).
Hi @tihomir No luck with this
[{"eventId":"1","eventTime":{"seconds":"1678495511","nanos":784316332},"eventType":"EVENT_TYPE_WORKFLOW_EXECUTION_STARTED","version":"0","taskId":"6305735","workflowExecutionStartedEventAttributes":{"workflowType":{"name":"IntegrationWorkflow"},"parentWorkflowNamespace":"","parentInitiatedEventId":"0","taskQueue":{"name":"IntegrationWorkflowTaskQueue","kind":"TASK_QUEUE_KIND_NORMAL"},"input":{"payloads":[{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"MzE="},{"metadata":{"encoding":"anNvbi9wbGFpbg=="},"data":"ImE2MTE5ZDYxLTQ2Y2EtNDc5OS1hOWYyLWFmN2ZjNGZlMzY0ZCI="}]},"workflowExecutionTimeout":{"seconds":"0","nanos":0},"workflowRunTimeout":{"seconds":"300","nanos":0},"workflowTaskTimeout":{"seconds":"10","nanos":0},"continuedExecutionRunId":"6105d13f-d850-4afd-9dbd-2d606d8d6bd0","initiator":"CONTINUE_AS_NEW_INITIATOR_CRON_SCHEDULE","continuedFailure":{"message":"workflow timeout","source":"Server","stackTrace":"","timeoutFailureInfo":{"timeoutType":"TIMEOUT_TYPE_START_TO_CLOSE"},"failureInfo":"timeoutFailureInfo"},"originalExecutionRunId":"9a9e9a51-dde0-429f-9fc9-bb508c62abfb","identity":"","firstExecutionRunId":"ca28ec90-729e-4a8f-b37c-08b448b0a2d2","attempt":1,"cronSchedule":"@every 60s","firstWorkflowTaskBackoff":{"seconds":"60","nanos":0},"prevAutoResetPoints":{"points":[]},"header":{"fields":{}}},"attributes":"workflowExecutionStartedEventAttributes"},{"eventId":"2","eventTime":{"seconds":"1678495571","nanos":789024184},"eventType":"EVENT_TYPE_WORKFLOW_TASK_SCHEDULED","version":"0","taskId":"6305743","workflowTaskScheduledEventAttributes":{"taskQueue":{"name":"IntegrationWorkflowTaskQueue","kind":"TASK_QUEUE_KIND_NORMAL"},"startToCloseTimeout":{"seconds":"10","nanos":0},"attempt":1},"attributes":"workflowTaskScheduledEventAttributes"}]
Hi @tihomir
No luck with the above suggestion. I tried to add this code on boot up and it did not work. Is there a way to configure firstWorkflowTaskBackoff? If i increase firstWorkflowTaskBackoff, will it fix the issue since my cron is scheduled every 60sec?
After you tried, do you see a WorkflowExecutionSignalled event in that executions history? Did you get a client error when you tried it?
This seems to be some issue on the server end as to why it’s not for some reason scheduling the first workflow task. One thing you could do is terminate this execution and start the cron again, but was hoping the signalWithStart could unblock it.
signalWithStart failed with an exception
WorkflowStub stub = workflowClientAccessor.getWorkflowClient().newUntypedWorkflowStub("workflowId",
"runId", Optional.empty());
WorkflowExecution startOnBootup = stub.signalWithStart("startOnBootup", new Object[]{}, new Object[]{});
2023-Mar-11 01:22:11.801 ERROR [main] o.s.b.SpringApplication - Application run failed
java.lang.IllegalStateException: Required parameter WorkflowOptions is missing
at io.temporal.internal.sync.WorkflowStubImpl.signalWithStart(WorkflowStubImpl.java:200)
Try maybe:
workflowClientAccessor.getWorkflowClient().newUntypedWorkflowStub(workflowId)
.signalWithStart("startOnBootup", new Object[]{}, new Object[]{});
workflowId should be the workflow id of your running execution.
@tihomir Looks like it was an issue with my code. Looks like workflow initialization was done on demand, but not on bootup. I moved the workflow start code inside the application-ready event listener and it worked as expected.
1 Like