Activity return to workflow took 30 milliseconds

Community Support
1

Hi All,

Am working on workflow optimization of my workflow execution, in that i have found that Activity took around 30 milliseconds to return the result back to workflow.Is this expected?
Example code
Workflow

logger.info("[CALL] Activity Started {}", System.currentTimeMillis());
      CallActivity callActivity =
          Workflow.newActivityStub(CallActivity.class, activityOptions);

      //Make external call
      String response = callActivity.call(stepName, translatedCall, topParentFlowId);
     logger.info("[Workflow] Total time took rest activity {}", total);

Activity:

 @Override
    public String call(String stepName, Call call, String workflowId)
        throws CheckedExternalException {
        long start = System.currentTimeMillis();
        logger.info("[CALL Activity] Activity started {}", workflowId);
        String result = callAsync(call, workflowId);
        long end = System.currentTimeMillis();
        long total = end - start;
        logger.info("[CALL Activity] Activity completed {}", total);
        return result;
    }

My Sample log:

[CALL Activity] Activity started 
[CALL Activity] Activity completed 169
[Workflow] Total time took rest activity 201

As per my log time taken to retrun back to workflow 32 milliseconds, Is this expected?

3

Hi, Temporal provides both SDK and server metrics as well as support for tracing which would give you a much better insight of your performances rather than using system outs.

Check out the sdk metrics and tracing samples, and worker tuning guide in docs here.

For sdk metrics look at

workflow_task_schedule_to_start_latency
activity_schedule_to_start_latency
workflow_endtoend_latency
activity_succeed_endtoend_latency

From server metrics you can measure

service_latency
persistence_latency
service_latency_userlatency
service_latency_nouserlatency

Note the overall workflow exec (end to end latencies) will depend on things like network latencies betwen your workers and service, persistence latencies, different types of service latencies etc. These things need to be taken into account if asking if certain execution time is expected or not.

Would start with tracing and sdk metrics to get an overall picture of workflow and each individual activity invocations and workflow / activity task latencies and then move to server metrics, first check persistence latencies and go from there.

4

Thanks for the reply, I have checked the Persistence latency

Problem:
I have Workflow with one Activity that makes an external rest call. I have calculated the external timing etc. once the response is back to the activity. the handover from activity to the workflow I see 120 milliseconds diff.

Setup details:
Running the Temporal server and my application K8 under the same cluster and using Postgres AWS RDS

Please find below the image of Persistence latency for the Last 7 days

image
image1594×1112 220 KB

Persistence latency UpdateTaskqueue

image
image1649×1142 138 KB

Can you please suggest, which operation should be monitored? is there documentation on the operation shown in the chart?
Eg: what is CompleteTasksLessThan?