Activities going to pending status and workflow execution is completed when using custom data converter

I am using Dynamic workflow approach where I am reading activities configuration from the config file(.yml), My custom config file has 2 parallel branches, 1st branch make 3 micro service calls and second branch makes 5 micro service calls.

Parent workflow calls child workflows based on actor count which comes from input and based on count that many child work flows gets executed in parallel and each child workflow executes the activities based on custom config file.

I am using custome Data converter to encrypt/decrypt payload at Temporal client level.

Whenever I enable my custom data converter atleast one activity call goes to pending status and never gets completed and workflow execution gets completed.

Below is the only error I see from the logs:
2024/03/04 10:48:43 INFO Task processing failed with error Namespace default TaskQueue idp-call-taskqueue WorkerID 1628@5CG2142KHJ@ WorkerType ActivityWorker Error workflow execution already completed

This happens only when I use my custom data converter, if I use default one I get valid response without any error.
Also I tried reducing number of parallel activity calls from 5 to 3 then I get valid response irrespective of custom data converter.

I tried increasing activity start to close timeout but with no luck, currently this is set to 50000 ms
Also tried configuring heartbeattimeout, currently this is set to 5000 ms

Could someone please help me figure out this issue which is currently my blocker, if I am missing any thing ?
Let me know if you need me to share any of my code snippet which I would be happy to do.Thanks.

This issue has been resolved, actually I was getting decode payload exception from one of the microservice call which was the root cause for activities going into pending status, After fixing the decode issue, My activities are executed as expected without any issues.Thanks.

1 Like