Send 2000 events per second to 20 different types of workflows

Community Support
,
1

Hi!

We have

  1. 20 different types of workflows
  2. need to process 2000 events in seconds
  3. each workflow type waits for each event

What is the best way to implement?

  1. send each signal to each workflow type, something like:
    send to workflow-id=“type1-userId-5”:
    send to workflow-id=“type2-userId-5”:
    send to workflow-id=“type3-userId-5”:
    …
    send to workflow-id=“type20-userId-5”:

it means, send 2000*20 signals every second

  1. send batch, something like:
    temporal workflow signal
    –name NewUserTransactionSignal
    –input ‘{“Input”: “As-JSON”}’
    –query 'ExecutionStatus = “Running” AND (WorkflowId=“type1-userId-5” OR WorkflowId=“type2-userId-5” OR WorkflowId=“type3-userId-5” …type20-userId-5) ’ \

it means, send 2000 batches per second

  1. some other one?

Thanks!

2

It is difficult to give recommendations without understanding the use case. Are you trying to send 2,000 signals per second to each workflow instance?

3

I’ll try to explain

We have events, it’s different users actions.
Like userN do action-1, userN do action-2, userN do action-3, etc…

In total, there are now about 2000 events per second and 95% of all events are action-3

In the future we are planning a large number of different workflows, like

  1. Workflow “UserBonus”
    start trigger: action-1
    await signals: action-3, action-4, action-5

  2. Workflow “UserBlocker”
    start trigger - action 2
    await signals - action-3, action-6, action-7

  3. Workflow “UserNotifier”
    start trigger - action 2
    await signals - action-3, action-7, action-8

  4. Etc…

each workflow runs for each user, like
workflow type: UserBonus
workflow name: workflow-user-bonus-for-user-id-N

workflow type: UserBlocker
workflow name: workflow-user-blocker-for-user-id-N

action-3 necessary for all workflows types, when user do action-3 we need to send a signal to all 20 types of workflows

so each new workflow type will add ~2000 requests(signals) per second to Temporal

Screenshot 2024-08-20 at 21.39.00
Screenshot 2024-08-20 at 21.39.001666×1040 82.3 KB

4

Thanks for the explanation. Processing 40k requests per second is possible, but it would require a pretty beefy cluster.

I recommend having a single workflow per user and implementing each “business workflow” as a function/class inside that workflow. This way, adding a new “business workflow” will not increase the load on the cluster.

At Uber, we had a similar use case where the number of “business workflows” was up to 100 per user. The incoming stream had up to 10k requests per second. Back then, we couldn’t process 1 million requests per second. However, collocating all these 100 “business workflows” inside a per-user Temporal workflow allowed us to support the load.

5

Thanks for the answer

Does this mean that it is always a running single workflow for each user via Continue-As-New?

When we change the version of a workflow, can we send signals “do Continue-As-New” to all running workflows, it is ok for Temporal?

6

You don’t need to run a workflow for a user that has no “business workflows” running at the time. You can use signalWithStart to start the user workflow on demand.

You could use signals for this. But exiting user workflow when there are not running workflows is simpler IMHO.