Need to understand how Threads work with temporal

Community Support
1

Hello ,
We have started working with Temporal with spring
our flow goes like this :
controller → service → temporal WorkflowImpl ->ActivityImpl → service

so if we set some value in ThreadLocal at controller layer , still we could get the same at ActivityImpl
how is it possible?
Could you please help us understanding how threads work ? whether at workflowImpl level new threads are created ? As while logging we could see that thread names are different at service layer and workflow layers.

One more question :
How can I invoke this workflow synchronously?

WorkflowStub stub = workflowClient.newUntypedWorkflowStub("createUser", options);
stub.start(user , null, null);

below code works as expected , but I need to invoke untyped workflow synchronously

 HelloWorldWorkflow workflow = client.newWorkflowStub(HelloWorldWorkflow.class, options);
        // Synchronously execute the Workflow and wait for the response.
        String greeting = workflow.getGreeting("World");
1 Like
2

Regarding workflowClient.newUntypedWorkflowStub
right after you start the workflow you can call
stub.getResult(...);
getResult will perform a long poll on the server waiting for the workflow completion event.

For the thread questions, within your workflow code you should use io.temporal.workflow.WorkflowLocal or io.temporal.workflow.WorkflowThreadLocal
Activities have to be thread safe as you are registering a single activity instance with the worker.

3

Thanks so much @tihomir
Is there doc I can refer to understand the threads mechanism better?
threads can become a major issue for us if we don’t have much clarity as we are using spring with temporal Java-sdk

4

Take a look at Workflows in Java | Temporal documentation
Definitely need a similar section for Activities and will work on it.
Let me know what you think is missing there or can be improved.

5

Such context propagation is possible through interceptors. See how opentracing support which is essentially context propagation is implemented. It is 100% interceptor based.

6

@maxim
Actually even without doing anything still we are getting the values at activityImpl from ThreadLocal, is it expected ?
As the threads are different . Please correct me if I am missing anything.

7

I doubt. The threads used by activities are owned by a separate thread pool.

8

Thanks @maxim ,
even we had this doubt too,

is there a way we can execute entire workflow in one thread ?
Or something which can fix the below issue ?

flow :

controller -> workflow[impl & activity]  -> service
<< controller calls workflow stubs>>
<<service bean is injected in workflow using dependency injection>>

request 1 sets value xyz in ThreadLocal , at service we get xyz [ from inheritableThread values]
request 2 sets value abc in ThreadLocal BUT here at service we are still getting “xyz” and not “abc”

Does temporal provides any way that can fix our issue.
Considering we are extensively relying on the threadLocal values at service layer, it has become a blocker for us.

9

Does temporal provides any way that can fix our issue.

Yes, you have to use interceptors or ContextPropagator to pass your context between all the threads and the headers.

10

ok @maxim ,
how to get the values from context Propagator ?
is it based on implementation where if MDC is used , then MDC.get(“key”)
or
Temporal java-sdk exposes some mechanism/api to access it from workflow/activity classes?

11

You can implement a custom context propagator by implementing the ContextPropagator interface, see sample here: sdk-java/ContextPropagationTest.java at master · temporalio/sdk-java · GitHub
and also look at the tests there they help.

1 Like
12

Hi @tihomir ,
Thanks for the prompt response.
I have a few queries wrt to threads and workers . We are analyzing one performance issues.

  1. what is the size of task queues ? Can we configure it ?

  2. How many threads can a worker execute [ size of thread pool ] ? What happens if all the threads are executing?

  3. What happens if a worker fails to pick task from task queue or it is shot down? Will the workflow timeout after some time?

  4. What happens if workers instance shuts down while processing an activity?

  5. How to find which activity took how much time without Temporal UI , we have cli access.
    I have response from this cmd -
    tctl --namespace xxx workflow show -w workfloId -r runId

13

  1. Task queues can be treated like distributed queues, managed by the Temporal service. Their size is also managed by the service (can grow dynamically with the number of tasks they contain). There are no configuration options for size currently.

  2. Max number of threads for all workers created by a WorkflowFactory is configurable via WorkflowFactoryOptions.maxWorkflowThreadCount. Default is 600 threads.
    Workers have a cashing mechanism for workflows. Whenever there are no more available threads, a worker can evict a cashed workflow for other workflows to execute/continue execution.
    Check out this forum thread for worker configurations, as well as this documentation PR.

1 Like
14

  1. Your workflow code is executed when a worker receives a workflow task. Workflow task has a default timeout of 10 seconds. If your worker instance shuts down it does not report back to the service. This will cause a workflow task timeout (you can see a WorkflowTaskTimedOut event recorded in history). On the service side, the timed out task is retried to ensure that workflow execution can continue when your worker comes back up.

  2. Would suggest watching this video on activity timeouts.

1 Like
15
  1. Try adding --print_datetime flag to see event timestamps.
1 Like
16

Thanks @maxim @tihomir

17

Just wanted to add a little for question 5

you can use the activity_succeed_endtoend_latency SDK metric to measure total execution time (including retries) for an activity that successfully completed. there isn’t yet a metric that can be used for failed activity invocations but should be added to future SDK releases.

1 Like