Best way to getHeartBeatDetails inside a workflow

Community Support
1

Hello All,

I am using the Go-sdk.

Here is a bit of a context to the question being asked.

  1. I have a workflow that uses task queues specific for each activity that the workflow schedules.
  2. The workflow uses a selector to see which if these activities are completing/returning with errors etc
  3. Within the activity, I record the heartbeat in regular intervals.
  4. When the workflow hears back from the activity as a successful execution, I use the Activity response to propagate the information from the activity back to the workflow (which it then uses for a subsequent ContinueAsNew)
  5. However when the activity fails and I would like to get the last recorded Heartbeat from the activity.
  6. I see there are APIs for getting the Heartbeat.
activity.GetHeartbeatDetails()

and there is

client.DescribeWorkflowExecution()

But both these methods have context.Context as their parameter. However I only have workflow.Context and the context that was passed to workflow.ExecuteActivity (the later is just built using the activity specific task queue name).

The question I have is:
What is the value that can be passed as a parameter to the GetHeartBeatDetails() method to get the last known HeartBeat from that particular activity.

Regards,
Ananth

2
  1. However when the activity fails and I would like to get the last recorded Heartbeat from the activity.

When an activity fails a workflow receives an ActivityError that wraps a TimeoutError. Use TimeoutError.LastHeartbeatDetails and TimeoutError.HasLastHeartbeatDetails to extract the last heartbeat details.

  1. I have a workflow that uses task queues specific for each activity that the workflow schedules.

This is rarely a good idea. We recommend using a single task queue for all activities unless you need to specify individual rate limits for those activities.

3

Thanks a lot for the response @maxim . I will alter the design per your advice.

I initially assumed that the activity.GetHeartBeatDetails() when run from the a new run of the activity code would be returning back the last runs state of the heartbeat in the new activity instance process. I was wrong in that understanding. It makes sense to get the last known heartbeat details either at the workflow level where the activity is being managed or external to the workflow. Because of my initial misunderstanding I designed a per activity task queue and that activity had to resume by calling activity.GetHeartBeatDetails except that it was not getting any (as the new activity context is different from the old one maybe ? )

Thanks for the advice and really appreciate the effort you are putting into the community forum. Worked with airflow and prefect before and can clearly see how temporal is a differentiator.

4

I initially assumed that the activity.GetHeartBeatDetails() when run from the a new run of the activity code would be returning back the last runs state of the heartbeat in the new activity instance process.

This is the correct assumption. When an activity is retried it can retrieve the value of the last heartbeat from the failed activity using activity.GetHeartBeatDetails(). And activity indeed can resume from where the previous activity attempt left it off.

I assumed that you wanted to get the heartbeat value from the workflow code. In this case relying on the error is the way to go.

I’m not sure how per activity task queue is related to heartbeating/retries at all.

5 
I’m not sure how per activity task queue is related to heartbeating/retries at all.

Here is what I was thinking and why I went with a per activity task queue.

  • The application that is being built is a workflow that would read from pulsar and pump data into a database.
  • The workflow would schedule as many activities as the number of partitions for the pulsar partitioned topic.
  • The heartbeat details payload maintained the partition name and the message offset Id to resume stream processing from when the activity starts.
  • I wanted a mechanism wherein the activity would resume from the last point of exit. By exit, I mean either a graceful termination or an error state.
  • In the case of error exit, the activity would have access to the heartbeat details
  • In case of a normal exit, what I realised was GetHearbeatDetails did not return any value in the next run of the activity.
  • The reason why I thought per activity task Q made sense was because activity could manage itself. My initial assumption was that a task queue is sticky to the heartbeat details it manages. This was my misunderstanding. Perhaps what I missed was that the heartbeat details would be attached for a retry attempt automatically.
  • Thus the initial workflow code maintained one task Q for each activity and the workflow did not have any major application logic.

I have subsequently changed the above design to manage more centrally and now have a common task queue for all the workers.

Regards,
Ananth

6

There is no “next run of the activity” in case of normal exit as the activity is considered completed at this point. An activity can return any information to the workflow if necessary as part of its result.

  • The reason why I thought per activity task Q made sense was because activity could manage itself. My initial assumption was that a task queue is sticky to the heartbeat details it manages. This was my misunderstanding. Perhaps what I missed was that the heartbeat details would be attached for a retry attempt automatically.

Correct. Task queues are not related to activity heartbeat/retries in any way.