Worker processing workflow architecture question

Community Support
, ,
1

Good evening to everyone!
I have an architecture question that bothers me before we take Temporal as the main framework for our project.

Some details:
Imagine we take the flow, described in this Quick Start for Java tutorial

Workflow implementation is simple, but we change it a bit

 @Override
    public void transfer(String fromAccountId, String toAccountId, String referenceId, double amount) {
        System.out.println("Start workflow");
        for (int i=0;i<=1;i++) {
            System.out.println("start i="+i);
            account.withdraw(toAccountId, referenceId, amount);
            System.out.println("middle i="+i);
            account.deposit(fromAccountId, referenceId, amount);
            System.out.println("end i="+i);
        }
    }

Script output

Start iterations
start i=0
Withdrawing $18.740000 from account 002-002. 
middle i=0
Depositing $18.740000 into account 001-001. 
end i=0
start i=1
Withdrawing $18.740000 from account 002-002. 
middle i=1
Depositing $18.740000 into account 001-001. 
end i=1

The main change is that we added the loop here. After that we start workflow in synch mode. Im running workers and server on the same local machine.
Now, according to the theory (for example here What is a Temporal Workflow? | Temporal Documentation)

  1. Temporal server will put the message into Workflow queue. (WorkflowExecutionStarted, WorkflowTaskScheduled events)
  2. appropriate Temporal Workere Entity will read the message and start the Workflow code, in our case “transfer” method above. (WorkflowTaskStarted event)
  3. We’ll see the “Start workflow” message in console, i=0, the message “start i=0”
  4. The Worker will stop workflow execution at activity “account.withdraw”, put the Activity message to the Activity queue (WorkflowTaskCompleted, ActivityTaskScheduled events)
  5. The puller will take the Activity message and execute Activity (it can be another node in workers cluster) (ActivityTaskStarted, ActivityTaskCompleted events)
  6. A Workflow worker continues workflow and runs deposit method.
    Questions
    a) I can see in the history the event “WorkflowTaskScheduled” after the step 5. Which component raises this event?
    I suppose it raises the worker, executing withdraw activity. After the activity execution puts the message to the workflow queue so other worker can execute workflow further. Is it correct?
    b) What happens with the Workflow Worker Entity process when it comes to the activity method and pushes the message to the activity queue, after WorkflowTaskCompleted event (step 4)? is it sleeps or stops somehow?
    b) Which worker executes next step between withdraw and deposit activity (step 6)? Is it the same worker, executed step 3? Or it can be a worker in other node of workers cluster?
    c) Is the new workflow worker (step 6 worker) runs the “transfer” method from the start? If so, why I can’t see the “Start workflow” message in console again?
    d) Judging by the messages in console, all the Workflow steps between activities are executed by the same thread and it waits while activities finish. I can see loop messages where i iterates 0,1,2 (see script output).
    Is it correct? if so, it looks really unstable, as the node running workflow can fail.

Unfortunatelly I cant’t merge my view of the Temporal architecture and the script output, it breaks my mind)) Would really appreciate your help

3

a) I can see in the history the event “WorkflowTaskScheduled” after the step 5. Which component raises this event?

All history events are created and persisted durably by the service. Once your activity worker responds back to service with RespondActivityTaskCompleted api that includes command for activity completion, service creates and writes the ActivityTaskStarted and ActivityTaskCompleted events in event history. At this point it needs to notify your workflow worker about activity completion and it does that by creating a new workflow task (WorkflowTaskScheduled event) that contains this completion info. When this workflow task is dispatched from matching service to frontend service and then to one of your workflow worker pollers service writes the WorkflowTaskStarted event into history.

4

b) What happens with the Workflow Worker Entity process when it comes to the activity method and pushes the message to the activity queue, after WorkflowTaskCompleted event (step 4)? is it sleeps or stops somehow?

Your worker will block the workflow thread and wait for service to deliver activity completion/failure info from service via a workflow task. This thread is also cached by worker in memory. Note that worker can evict this thread if needed to allow other executions to make progress (or when you restart workers the cache is gone). If this cached workflow thread gets evicted from cache while its waiting on activity completion/failure, when new event comes in (activity completion) worker would fetch full event history of this execution from service and replay it, putting it in same state as it was before and then continue (unblock that like one code and continue running your workflow code until its blocked again by temporal api).

5

b) Which worker executes next step between withdraw and deposit activity (step 6)? Is it the same worker, executed step 3? Or it can be a worker in other node of workers cluster?

Workflow executions are not tied to a specific worker in Temporal, however the service will try to give any updates (workflow tasks) to the same worker that was so far processing this workflow execution. This has to do with previously mentioned worker cache to reduce the number of times worker needs to replay event history for executions. Note that if your workflow worker goes down between these two steps the next task could be picked up by a different worker, would not have this execution cached and would then replay event history and continue its execution.

6

c) Is the new workflow worker (step 6 worker) runs the “transfer” method from the start? If so, why I can’t see the “Start workflow” message in console again?

If your workflow execution starts on worker A and then continues on worker B at some point you would see the “Start workflow” log again on that worker (if you use System.out.println which you should not, use workflow logger instead which is replay-safe). If it continues on same worker and it has it in its worker cache worker would not replay it from beginning.

Maybe video here could help understand replay a bit better.

Also videos here and here.

7

d) Judging by the messages in console, all the Workflow steps between activities are executed by the same thread and it waits while activities finish. I can see loop messages where i iterates 0,1,2 (see script output).

I would not use System.out in your workflow code, use workflow logger instead, which is replay-safe.
Someting like

public static final Logger logger = Workflow.getLogger(MyWorkflowImpl.class);

Using System out can be confusing at times as your logs would be written during replay as well.