Future Registered On Select in Update Handler Lost After Worker Restart

Community Support
1

I have an update handler setup. After trimming it down a bit it essentially does the following

	err = workflow.SetUpdateHandlerWithOptions(ctx, "StartChildWorkflow",
		func(ctx workflow.Context, req StartWorkflowRequest) error {
			futr := workflow.ExecuteChildWorkflow(ctx, req.WorkflowType, req.inputs...)

			workflowState.MainLoop.AddFuture(futr, func(f workflow.Future) {
				// todo if the worker instance is reset, this handler never fires

				err := f.Get(ctx, nil)
				if err != nil {
					handleChildWorkflowErr(ctx, err)
				}

				workflowState.clearWorkflow()
			})

			workflowState.RunningWorkflow = true
			workflowState.RunningWorkflowName = "dry-run"

			workflowState.NumberOfCalls++

			return nil
		},
		...

Where it starts a child workflow and adds its future to a selector. It works as expected under normal conditions. But if the worker is restarted while the child workflow is still running, the future never fires in the parent workflow.

A similar function (with no errors being returned) works in a signal handler.

workflowState.MainLoop.AddReceive(workflow.GetSignalChannel(ctx, "test"), func(c workflow.ReceiveChannel, more bool) {

Go 1.22.5
Temporal SDK 1.26.1 sdk module - go.temporal.io/sdk - Go Packages
Temporal Server 1.24.1 from the temporal cli

2

I tried to replicate the scenario you described and I don’t experience any issue after restarting the worker. Can you provided a stand alone reproduction on the latest Go SDK release?

3

Thanks for checking on it.

Updated to latest. 1.28.1 workflow package - go.temporal.io/sdk/workflow - Go Packages

paired it down a bit further to this.

func HelloUpdates(ctx workflow.Context) error {
	readyToDie := false
	testState := 5
        mainLoop := workflow.NewSelector(ctx)
	err = workflow.SetQueryHandler(ctx, "test_state", func() (int, error) {
		return testState, nil
	})
	if err != nil {
		return errors.WithStack(err)
	}

	err = workflow.SetUpdateHandlerWithOptions(ctx, "sweet-update",
		func(ctx workflow.Context) error {

			fut := workflow.ExecuteChildWorkflow(ctx, HelloWorkflow)

			mainLoop .AddFuture(fut, func(f workflow.Future) {
				// if the instance is reset, this handler never fires

				testState = 0
			})

			return nil
		},
		workflow.UpdateHandlerOptions{},
	)
	if err != nil {
		return errors.WithStack(err)
	}

	for !readyToDie {
		mainLoop .Select(ctx)
	}

	return nil
}

func HelloWorkflow(ctx workflow.Context) error {
	_ = workflow.Sleep(ctx, 30*time.Second)
	return nil
}

still not getting that state updated to 0 if I reset the worker in the middle.

4

actually, now I’ve done something where it doesn’t update the state even with the worker left running. Let me dig.

5

FWIW running this workflow, sending one update, then restarting the worker while the child workflow is running, after 30s the callback is called and testState is updated.

6

Ok thanks for checking. Hmm. I guess at least I know the concept should work. Running that same example exactly I’m not getting the testState update.

Confirming running v1.28.1 of the sdk and temporal cli v1.0.0 with server Server 1.24.2 run command temporal server start-dev --dynamic-config-value=frontend.enableUpdateWorkflowExecution=true

7

Well. I’ve tried a few things, not sure why it isn’t working for me. I think for now I’ll just skip using the select statement and rewrite the handler using a temporal goroutine.

		func(ctx workflow.Context) error {
			fut := workflow.ExecuteChildWorkflow(ctx, HelloWorkflow)

			fmt.Println("handler firing")
			fmt.Printf("%v handler firing\n", fut)

			workflow.Go(ctx, func(ctx workflow.Context) {
				_ = fut.Get(ctx, nil)
				fmt.Println("finally")

				testState = 5
			})

			return nil
		},