Hello,
I have use case to build a data transformation. I’ve multiple sources of kafka as incoming data, and input data will be transformed into a target data model. Finally target data model will be transmit to a downstream system via kafka topic.
In between data ingestion and transformation user can add filtering and validations on the incoming data.
User can defined the workflow choosing source, filtering, validation, transformation and transmission.
- I wanted to understand how temporal can help in this use-case to orchestrate the tasks?
- How can we implement a kafka consumer to read the incoming data and orchestrate remaining tasks?
- Each workflow defined by the user needs to run on a separate thread. E.g. Workflow 1 - Reads data from source ‘A’ → filtering → validations → transform into a target data model (A1) → transmit to downstream (A2). Workflow 2 - Reads data from source ‘B’ → transform into a target data model (B1) → transmit to downstream (B2). Both workflows must be running on two different threads.
Appreciate someone can advice on how temporal help to implement above. Thanks.