We have a workflow with a decision task that has a timer of 10 secs. After 10 seconds we start an activity which does a simple API call. The activity is taking up to 5-10 mins (in some cases) to be executed if we run 3000 such instances, using 20 workers and a 3 node cassadra with 4 cores (peaks at around 30% CPU). None of the cadence services or workers have CPU usage more than 0.1 cores.
Can you please help us point out the performance bottlemeck or incorrect config?
Here’s the execution events - https://pastebin.com/bd4SKJPJ