Activity in scheduled state for very long

Community Support
,
1

Hello,

We have a workflow with a decision task that has a timer of 10 secs. After 10 seconds we start an activity which does a simple API call. The activity is taking up to 5-10 mins (in some cases) to be executed if we run 3000 such instances, using 20 workers and a 3 node cassadra with 4 cores (peaks at around 30% CPU). None of the cadence services or workers have CPU usage more than 0.1 cores.

Can you please help us point out the performance bottlemeck or incorrect config?

Screenshot 2020-09-16 at 10.32.30 PM
Screenshot 2020-09-16 at 10.32.30 PM1437×734 144 KB

Here’s the execution events - https://pastebin.com/bd4SKJPJ

Best,
Shubham.

1 Like
3

My guess is that the workflow task queue is a bottleneck. By default in Cadence, it runs in a single partition only. You can increase the number of partitions for the task queue using dynamic config.

I’m not sure about the Cadence configuration file format. Here is how we specify it in Temporal:

4

Thanks a lot @maxim. We’ll try to find this config and set it.

Really wish temporal was stable so that we could start using it :slight_smile:

5

Almost there :slight_smile:

1 Like