Most of my workfows are long running workflow and they either sleep for a specific period or await for a signal/trigger.
When i create workflows , the workflow immediately enters sleep or await after doing little bit of bootstrapping. If i submit 600 such workflows i see each of this wait/sleep is actually backed by a thread from (workflow method pool ) i.e. almost 600 threads.
and i see all the threads are waiting notification
"workflow-method": awaiting notification on [0x00000000f165ea08], holding [0x00000000f165ed58] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2044) at io.temporal.internal.sync.WorkflowThreadContext.yield(WorkflowThreadContext.java:83) at io.temporal.internal.sync.WorkflowThreadImpl.yield(WorkflowThreadImpl.java:410) at io.temporal.internal.sync.WorkflowThread.await(WorkflowThread.java:45) at io.temporal.internal.sync.SyncWorkflowContext.await(SyncWorkflowContext.java:697) at io.temporal.internal.sync.WorkflowInternal.await(WorkflowInternal.java:309) at io.temporal.workflow.Workflow.await(Workflow.java:768)
I just see huge number of threads, but really dont see any issues or anything going wrong…
Is this normal?
Also would large number of threads cause cpu trashing etc?
How should i size my worker/java process?
Apart from huge number threads waiting on notification, i dont really see anything else going wrong per say.
However, if i restart the process, the thread level do come down significantly ( i am trying this on 1.0.5.snapshot) which alredy has a fix for dead lock issue
and workflow still continues as normal, is this because the sticky cache in worker? i seee the stick cache size in sdk dashboard grows over time.
E.g when i create 600 worflows i see
When i restart the worker i see
Overall my sticky cache in sdk dashboard looks like
Any tweaking required in confs or Service/Client Options etc?