Workflow Activity gets stuck intermittently

inishchith
Tue, 25 Mar 2025 10:11:36 +0000
Community Support

Hi Team,

For context, we have long running activities in temporal and we self-host the temporal cluster.

We’re observing 2 cases of workflow getting stuck intermittently and not progressing indefinitely

1. Here the workflow has gotten stuck, we see that the workflow execution has started and workflow task has been scheduled, but it stays stuck in this state indefinitely.

In the metrics dashboard, we observe that the Matching Service doesn’t receive the AddWorkflowTask from the History Service in this case, which I believe should be received.

2. Here the activities are stuck in PENDING_ACTIVITY_STATE_SCHEDULED

In the metrics dashboard, we observe that the Matching Service receives the the AddWorkflowTask, but doesn’t received the AddActivityTask from the History Service in this case. In case of a successful workflow, we observed that both the events are received accurately. We believe this isn’t related to any specific activity code as we see this happening across activities (long and short running).

Note:

We checked the resource utilization of all components in the temporal service, none of them seems to be exceeding 15%.

Temporal Service Version - v1.24.3

8 posts - 4 participants