.NET SDK Activity Cancellation Logic

Community Support
,
1

(Editor: you don’t appear to have a dotnet-sdk tag).

Imagine you have a long-running activity, and you want to stop it running when the parent is cancelled.

With the Java SDK: the intention is that you heartbeat to determine whether the parent has been cancelled or not. So, you heartbeat,and handle any exceptions of the ActivityCompletionException. We happen to wrap the interaction with a thread and supplier.

Now, to the .NET SDK. In the activity, you have access to ActivityExecutionContext.Context.CancellationToken (AEC).

So, this is obviously different to the Java implementation, in that C#'s async has an established cancellation pattern.

So, my question is this . . . (i) is there a separate thread to monitor the parent workflow’s cancellation status and then cancel the CancellationToken or (ii) does it work similarly to Java’s heartbeat implementation, i.e. you have to implement the heartbeat (ActivityExecutionContext.Current.Heartbeat()) before it can detect that the parent is cancelled?

I realize this is a subtle question, but it’s an important detail.

Note: I think you need a C# example that is “complete”, i.e. demonstrates (a) cancellation, (b) heartbeat, and (c) the sequence to handle both. The examples just heartbeat a progress %. What’s more useful, I think, is a threaded heartbeat that has to heartbeat within the heartbeat ttl specified on the parent.

Many thanks!

2

Hrmm, other posts have been able to set a dotnet-sdk tag, I have now set that tag on this post.

Correct, and matches Go (context.Context), TypeScript (CancellationScope), and Python (asyncio cancellation) in that heartbeat does not throw.

Are you asking whether the internal code has a separate thread or if you should? If whether we have that, we simply set this token as canceled if the server informs us it is canceled (which is based on internal throttled heartbeat but done in the background in our Rust layer, not inline with the heartbeat call). If whether you should, that is up to you. The cancellation token should be treated like any other .NET cancellation token you may receive from a framework

Under the hood, yes. You have to heartbeat to get server-sent cancellation (technically the token can get cancelled on worker shutdown too which is not server-sent).

I assume this sample is the one you’re referring to that could be improved? I think you can apply general .NET cancellation token approach not specific to Temporal, and maybe that’s using a background task to listen to cancel. Is this the sample that is needed, a general sample showing how to wait for .NET cancellation token in a background task?

3

Thanks for the reply. Yeah, debugged/played some more. You have to heartbeat for it to detect that the Activity is cancelled (totally reasonable), and it’ll cancel the CancellationToken.

Note: before reading the following, I don’t consider myself competent at C#'s Task/async patterns. But, it appears to work, and is what I was asking for.

Wrapping long-running code within an Activity with a periodic heartbeat . . .

return await ActivityHeartbeatUtils.withHeartbeat(
                Task.Run(() =>
                    CompareSchemasInternal(
                        request, ActivityExecutionContext.Current.CancellationToken)),
                TimeSpan.FromMilliseconds(1000));

The implementation . . .

public async static Task<T> withHeartbeat<T>(Task<T> task, TimeSpan timeSpan)
{
    while (!task.IsCompleted)
    {
        ActivityExecutionContext.Current.Heartbeat();
        await Task.WhenAny([task, Task.Delay(timeSpan)]);
    }
    return await task;
}

I will improve with a variation that has no Task result, and an interface to provide the heartbeat payload, etc.

BTW: we have the equivalent (proven) equivalent for Java (though it’s, by necessity, a little more complex). It might be useful to share, as it might actually be useful to SDK developers. We distributed it as part of an internal SDK.

4

Yes, many people do implement an “auto heartbeater” this way. We even have an issue open to support it officially. Be advised, we encourage heartbeats within the logic as opposed to the background for two reasons.

Firstly, a heartbeat in the background will only timeout on the server if the worker crashes hard, it won’t timeout if the activity is hung on some logic/code because background will still be running, whereas heartbeating within the logic will properly timeout.

Secondly, heartbeats can take details which are often used to mark progress so if an activity fails and is retried (due to heartbeat timeout or other reason), it can access the last heartbeat details to know how far it got. But you can only set meaningful details within the logic, not in the background.

5

But, if the heartbeat is done within logic, it’s hard to control the periodic heartbeat requirement, as dictated by the parent workflow.

I understand the general point though. In an example of ours, we might be migrating a database table’s contents from one database to another. Fortunately, those operations fail if, for example, database connections fail, and will error out – causing (desirable) retry.

Basically, your preference is for logic to frequently emit something to call heartbeat, and let the SDK throttle the heartbeat-to-server calls.

Thank you!

1 Like