How can I know my worker is busy enough?

Community Support
,
1

Hi, me again :slight_smile:

I have the following use case:

Given a list of .parquet files located on S3, my workflow processes each row and makes gRPC calls to an external service. I’m using a variation of the pattern used in the Sliding Window Example:

  1. My main workflow creates the partitions to be processed, using a configurable partition_size.
  2. For each partition, it spawns a child workflow that goes over the entire partition and spawns another child workflow for each row that does the actual reading, processing, and gRPC calling (the first two workflows only use offsets and limits).

For example, with 200 rows total and partition_size of 10, there would be 1 main workflow, 20 child workflows for partitions, and 200 child workflows processing individual rows.

The issue I’m struggling with is that there seems to be a bottleneck in the process, since the external gRPC service is getting requests at around 15 RPS. This is despite having around 200 workflows concurrently making gRPC calls.

I’m looking for leads on how and where I should look to find the bottleneck.

Thanks!

1 Like
2

can you share your worker options? how many worker processes do you have running supporting your fanout use case?
do you have service and sdk worker metrics available?

3

Hi, thanks for responding! :heart:

I recently started exploring the metrics after reading some past posts here, which were very helpful.

It turns out the issue was with my implementation. I had initially used the approach from the Sliding Window Batching example, where the “grandson” workflow responsible for the actual processing was implemented as a single activity containing all the logic. After splitting this activity into separate activities, I saw the performance boost I was looking for. This makes sense, as it gives Temporal more opportunities to parallelize the process.

I’m still new to Temporal, so this is all part of my learning curve! :blush: