I am using Dynamic workflow approach where I am reading activities configuration from the config file(.yml), My custom config file has 2 parallel branches, 1st branch make 3 micro service calls and second branch makes 5 micro service calls.
Parent workflow calls child workflows based on actor count which comes from input and based on count that many child work flows gets executed in parallel and each child workflow executes the activities based on custom config file.
I am using custome Data converter to encrypt/decrypt payload at Temporal client level.
Whenever I enable my custom data converter atleast one activity call goes to pending status and never gets completed and workflow execution gets completed.
Below is the only error I see from the logs:
2024/03/04 10:48:43 INFO Task processing failed with error Namespace default TaskQueue idp-call-taskqueue WorkerID 1628@5CG2142KHJ@ WorkerType ActivityWorker Error workflow execution already completed
This happens only when I use my custom data converter, if I use default one I get valid response without any error.
Also I tried reducing number of parallel activity calls from 5 to 3 then I get valid response irrespective of custom data converter.
I tried increasing activity start to close timeout but with no luck, currently this is set to 50000 ms
Also tried configuring heartbeattimeout, currently this is set to 5000 ms
Could someone please help me figure out this issue which is currently my blocker, if I am missing any thing ?
Let me know if you need me to share any of my code snippet which I would be happy to do.Thanks.