Problem with synchronisation between workflows

Community Support
,
1

Hello!
I’m a new in temporal apologize if it’s a duplicated topic
I’m considering using temporal for my project but I faced with one obstacle.

Context:
There is a group of workflows/applications:

  1. File processing workflow - parse the file, save records to DB and pass the processed record to another workflow (file could contain millions of records)

  2. Record processing workflow - process record with specific business logic and create or put to record group, each record group is stored in DB
    Record processing flow processes ONLY by one record

  3. Group processing workflow - process group with specific business logic, group processing can not start until all records are processed

Problem:
The problem is here that mechanism with trigger of group processing workflow is absent because group processing can not start until all records are processed.
I considered to use child workflows where parent workflow triggers above workflows.
However temporal has a limitation - 50k events per workflow but file can contain hundred thousands or millions of records.

Is it possible to overcome this problem with child workflows?
Maybe temporal has another orchestration mechanism for such cases?

2

Have a parent workflow that invokes:

  1. File processing workflow
  2. Record processing workflow which iterates over records using sliding window approach. See the sliding window batch example.
  3. Group processing workflow
3

I forgot to thank you!

We implemented sliding window for our batch processing and it seems this approach is fitted for us!

1 Like