After CancellationScope cancel() call, activity heartbeat call still successful

I run into an issue that after scope.cancel() is called, activity in the CancellableScope sometimes keeps on going until fully finished without throwing ActivityCompletionException.

Timing is weird and here are some context/experiments:

  1. Code is pretty much the same as HelloCancellationScopeWithTimer.java
  2. in Activity code if I sleep 1 sec between each heartbeat call it will keep going until my full loop which is 15 heartbeat calls
  3. Timer cancel is 3 seconds
  4. if under debug I wait for 2-3 seconds before invoking heartbeat(), ActivityCompletionException can be successfully thrown
  5. If I set sleep 5 secs between each heartbeat, ActivityCompletionException can be successfully thrown

Activity code:

    ActivityExecutionContext context = Activity.getExecutionContext();
    for (int i = 0; i < 15; i++) {
        sleep(1);
        try {
            context.heartbeat(i);
        } catch (ActivityCompletionException activityCompletionException) {
            LOG.info("Successfully cancelled and cleaned up");
            throw activityCompletionException;
        }
   }

Any thoughts or pointers?

Activity hertbeats are rate limited to avoid overloading the service. So not every heartbeat call results in a heartbeat request to the service. You can reduce the heartbeat timeout to force more frequent heartbeats.