No workflow timeout when signal deserialization error happens

Community Support
,
1

Hi,
I have a workflow, which sends the workload for 3rd party service to process and then waits for signal when processing is finished.
It is done in a similar way as in a background checks example. There is set up a channel for signal and timer for timeout.

type Result struct{
	CreatedAt time.Time,
	Value string,
	// other fields
}

func waitForResult(ctx workflow.Context, timeout time.Duration) (Result, error) {
	var response Result
	var err error

	s := workflow.NewSelector(ctx)

	ch := workflow.GetSignalChannel(ctx, "workload_finished_signal")

	s.AddReceive(ch, func(c workflow.ReceiveChannel, more bool) {
		c.Receive(ctx, &response)
		workflow.GetLogger(ctx).Debug("received workload signal")
	})

	s.AddFuture(workflow.NewTimer(ctx, timeout), func(f workflow.Future) {
		err = f.Get(ctx, nil)

		// treat timeout as failure
		workflow.GetLogger(ctx).Debug("workload timeout",)
                err = errors.New("timeout")
	})

	s.Select(ctx)

	return response, err
}

func DispatchWorkload(ctx workflow.Context, req DispatchWorkloadRequest) (Result, error) {
	err := sendWorkload(ctx, req)
	if err != nil {
		return Result{}, err
	}

	return waitForResult(ctx, 60 * time.Minute)
}

Most of the time it works as expected, but in some rare cases signal fails to deserialize and I get the error

ERROR Deserialization error. Corrupted signal received on channel

Last time the error showed up, because time.Time type field got corrupted somehow.

In such case, workflow continues run indefinitely, not completing neither on signal receive, nor on timeout.

Any advice how to handle such error?

2

I believe corrupted signals are ignored. Have you tried to send another non-corrupted signal to that workflow? BTW you can use Channel.ReceiveWithTimeout instead of selector here.

3

Yes, if I send second correct signal, it works as expected.

I just expected that in case with corrupted signal, the workflow would still stop on timeout, which doesn’t happen for some reason.

Will try ReceiveWithTimeout, thanks.