The more general question is: what is the best scenario to use local activity?
My specific question: we have a distributed workflow service where the workflows are running. Some activities’ input relies on the previous activity’s response. In this case, should we always use local activity for the ‘previous’ activity? Otherwise the following activity would not always get some input fields.
You can think about the local activity as a pure performance optimization. So, for workflow correctness, there is no difference between normal and local activities.
I’m not sure I understood your question. An activity can return any result (if it is of a supported size) and workflow can pass data from that result to the next activity. Something like:
ok maybe I understand how the worker works incorrectly:
say on pod A, workflow reaches here:
result1 = activityA() <----------
but the workerB on pod B picked up activityA and returned a result to Temporal server
Pod A will still be able to get the response (returned by workerB) from Temporal, correct?
Workflow is not linked to any specific process. If you have 10 pods it can be executing in any of them at any time. This is handled seamlessly by Temporal service and the SDKs. So you should never make any assumption about exact location of the workflow code.
Activity invocation can happen on any process that supports the given activity.
Temporal makes sure that all the communication and recovery in case of failures happens automatically. So your example will work without any problem across any number of pods.
I would recommend watching this video that goes over Temporal execution model. It talks about its predecessor Cadence, but the core concepts are exactly the same.