Meaning of worker task slots available metric

Hi! Could you please help me to understand these metrics?
It never goes below ~235. My worker options:

		MaxConcurrentWorkflowTaskPollers: 24,
		MaxConcurrentActivityTaskPollers: 16,

		MaxConcurrentWorkflowTaskExecutionSize:  256,
		MaxConcurrentLocalActivityExecutionSize: 256,
		MaxConcurrentActivityExecutionSize:      256,

Does it mean I have enough worker task slots, or does it indicate that I should decrease the max concurrent task pollers size for better utilization?

It would mean that you have excess worker task slots available which could change during your peak hours - does your maru load test reflect expected peak loads?

does it indicate that I should decrease the max concurrent task pollers size for better utilization

Would look at worker tuning guide for this, specifically here.

Also look at this forum post for detailed info on selecting poller counts.

Thanks for your reply!
I have read worker turning post a few times, but unfortunately, changing the pollers count didn’t give a significant effect in my case.

I stopped on this Maru scenario and performed tests with different configurations of the Temporal/Maru Workers setup:

{
    "steps": [{
        "count": 1000000,
        "ratePerSecond": 1000
    }],
    "workflow": {
        "name": "basic-workflow",
        "args": {
            "sequenceCount": 1
        }
    },
    "report": {
        "intervalInSeconds": 10
    }
}

For now, I am still trying to understand where to bottleneck to performance, at least to get 1k wf/per sec. But the main goal it’s getting a clear understanding of what to tune to achieve even more of the 1k wf/per sec , and it seems to me is a too difficult task(

According to the forum post:

The general rule of thumb is that you always want to have your poller waiting for tasks, and not having task backlog build up.

Does it mean the problem with configuration Maru workers if P95 poll latency starts to drop? They are not capable of handling all tasks, I need to play with Poller’s count ?

Also, I would like to ask about HearBeat latency (I attached the screenshot). Does this also indicate a performance problem with the Maru worker itself?

Were you ever able to find a proper resolution? I am in a similar situation running Maru tests and trying to understand how to get to 1k workflows/sec. Currently I am way under this number (~25 wf/sec with 3 activities). @tihomir could you weigh in if not?

Thanks!

| how to get to 1k workflows/sec
Hi, what does this mean in terms of actions/s for your use case?
(server metrics emit "action" counter metric that could look at and share during your load test).

Essentially we are trying to understand the performance metrics using Cassandra vs. MySQL. Currently we are running performance testing with Cassandra. Ideally, we would like to be able to maximize the number of state transitions per second (perhaps 1k wf/sec is a misleading goal). We have several grafana dashboards set up to try and gather information on how to proceed, but the queries are either not set up correctly or hard to interpret.




Can you provide any feedback on how to improve/understand these readings? Are there other tuning metrics I should be looking at?

Thanks

Double-tapping this :slight_smile:

Would appreciate any guidance on these performance metrics :slight_smile: