Description of the Issue: The execution thread becomes unresponsive/block after a new workflow is submitted to Temporal

Community Support
1 
Code snippet :

ExecutorService executor = executor = Executors.newFixedThreadPool(2);


// scheduler will trigger the run

//fetch all the new data from DB
           
forEach(data -> executor.submit(() -> initiateWorkflow(data)));

private void initiateWorkflow(String data) {

WorkflowOptions workflowOptions = WorkflowOptions.newBuilder()
    .setTaskQueue(taskQueue)
    .setWorkflowId(bookingRequest.getCorrelationId())
    .build();


    log.info(“starting the workflow for  data {}” , data);

    CustomWorkflow workflow = client.newWorkflowStub(CustomWorkflow.class, workflowOptions);
    WorkflowClient.start(workflow::initiateWorkflow, data);

    log.info("updated processing status as {} for data: {}", IN_PROGRESS.name(), data);

}

We encountered an issue with our executor service, where we defined two threads. Both threads became blocked, evident from the absence of subsequent logs in the workflow after it started. Consequently, we were unable to process any data for a span of two hours. The problem resolved itself after the pods were restarted.Upon inspecting the temporal in the temporal UI using the workflow ID, I observed that the workflow initiated successfully.

We are uncertain as to why our execution thread did not return after the workflow initiation, as indicated by the code snippet: WorkflowClient.start(workflow::initiateWorkflow, data) as we don’t see next set of log. Also We don’t see ay exception in logs .

Have you encountered a similar scenario, or could you provide insights into the possible issues here?

Any assistance would be greatly appreciated.