Polling in workflow vs. Activity?

Community Support
1

You can basically poll from the workflow or from the activity. I’ve seen examples for both. With a workflow, you set a local variable and loop until you get a signal. With an activity, you can basically poll with a while loop with the activity heartbeat and wait for a callback as well. Like it’s not clear to me when you would do it in the workflow or the activity.

1 Like
2

Polling from workflow is generally a bad idea. By polling, I mean calling some activity in a loop. The problem with this approach is that it adds new events to the history without doing any useful work. And in extreme cases when polling needs to take a long time can exceed the history size limit. So we recommend polling from an activity.

Waiting for a signal is a completely different story. Waiting for a signal doesn’t affect history size and doesn’t introduce any additional latency. So it is a pretty good pattern assuming that the signal can be sent by an external system.

I’m not sure why you called waiting for a signal “polling”. Did you mean Workflow.await statement? It actually doesn’t increase workflow history, but you are not expected to ever put any code that mutates workflow state in any way inside the condition callback. I’ve seen examples of putting activity invocations there, but it is not supported.

4 Likes
3

Thanks for responding!

I’m trying to model a CI/CD pipeline as a workflow. When the CI/CD workflow starts, it will trigger a child build workflow. A build can consist of 1 or more jobs.

You can look at a job as a unit of work and a new process. So if you have a build that consist of a tree like structure where there can be jobs that run sequentially or in parallel. To capture this, I also have a job workflow which is what the build workflow will trigger.

CI/CD Workflow --> Build Workflow --> Job1 Workflow
                                  --> Job2 Workflow
                                  --> Job3 Workflow

Within the job workflow, it will request the job be started and then poll for status. Once the status is complete, it will start the a different stage for publishing results.

Within the worker method

boolean isPublishedComplete = false;

@Override
public startWorkflow() {

    Activity activity = ....

    activity.requestJobStart();
    while(true) {
        Workflow.sleep(Duration.ofSeconds(5);
        int status = activity.getStatus();
        if(status == COMPLETE) {
            activity.publishResults();
            break;
       }
    }

    Workflow.await( () -> isPublishComplete);
}
  • I have another signal method that will get signaled when it’s complete
@Override
public void publishComplete() {
     this.isPublishComplete = true;
}

The other reason I wanted to break it into the three workflows is the separation of concerns; a CI/CD workflow, a build workflow, and a job workflow.

So if i’m understanding you correctly, this is not recommended because it will flood the history with new events; events that are not useful.

If I go with your recommendation, then I’m guessing the code would look like

The JobWorkflow would a start method, that triggers an activity that starts the job and as well as poll.

Job Workflow

boolean isPublishedComplete = false;

@Override
public startWorkflow() {

    Activity activity = ....

    activity.requestJobStart();
    activity.waitForJob();

    Workflow.await( () -> isPublishComplete);
}

Job Activity

private ExternalApiClient client;

@Override
public void requestJobStart() {
   // send request to start job 
}

@Override
public void waitForJob() {
    while(true) {
       int status = client.getStatus();
       if(status == COMPLETE) {
            break;
       }       
      Thread.sleep(3000) // every 3 sec 
       Activity.heartbeat(status);
    }
}

Question

  1. Is using Thread.sleep alright? I know that the workflow has a specific sleep method and wonder if it matters for the activity.
  2. I have a question about the heartbeat. I don’t think i’m using it correctly. So right now, the loop will poll every 3s. Internally, the activity will keep track of the status every 3 sec as well based on the loop. If getStatus() hangs for more than 3 seconds, then the heartbeat is missed. How does the activity know that the heartbeat should be a 3 seconds thou? What happens with the flow from there? From the doc, it looks like ActivityTimeoutException is thrown. Does that mean the workflow caller would need to catch the exception and then look at the details section of the to know what steps to take next.

Thanks!

1 Like
4

You are correct that the first version is an anti-pattern and your second version looks fine.

Is using Thread.sleep alright? I know that the workflow has a specific sleep method and wonder if it matters for the activity.

Yes, it is fine inside the activity code. The activity code doesn’t have any restrictions that apply to the workflow code.

How does the activity know that the heartbeat should be a 3 seconds thou?

The server knows about this through the heartbeat timeout that you can specify through ActivityOptions. For example, if your activity heartbeat timeout is 60 seconds then you can heartbeat every 3 seconds and it is OK for getStatus to take longer.

1 Like
5

In this case, let’s say I want to track the status i’m polling internally within the activity. If I have a UI and I want to query the status externally, would I just expose a query method in the workflow and the workflow would query the activityMethod which would just return the internal status of the activity. Do you see anything wrong with that approach?

6

The describe workflow execution API returns status of all currently running activities. The status include the value passed to the last heartbeat call. So UI can get this information without querying the workflow.

7

Doesn’t Thread.sleep inside activity steal the thread from the threadpool or is there any underlying mechanism that doesn’t exhaust the threadpool ?

8

Activities are internally executed by thread pool, you can configure its size via WorkerOptions->maxConcurrentActivityExecutionSize (default 200). So my understanding is yes, thread.sleep inside activity would pause execution of the current thread for that activity execution.

1 Like
9

You can also implement such polling activity asynchronously using manual activity completion.

1 Like
10

Looking at the api it’s hard to understand. Do we any samples using the manual activity completion ?

11
1 Like