Data Transmission Use-case (including kafka)

Community Support
1

Hello,

I have use case to build a data transformation. I’ve multiple sources of kafka as incoming data, and input data will be transformed into a target data model. Finally target data model will be transmit to a downstream system via kafka topic.

In between data ingestion and transformation user can add filtering and validations on the incoming data.

User can defined the workflow choosing source, filtering, validation, transformation and transmission.

  1. I wanted to understand how temporal can help in this use-case to orchestrate the tasks?
  2. How can we implement a kafka consumer to read the incoming data and orchestrate remaining tasks?
  3. Each workflow defined by the user needs to run on a separate thread. E.g. Workflow 1 - Reads data from source ‘A’ → filtering → validations → transform into a target data model (A1) → transmit to downstream (A2). Workflow 2 - Reads data from source ‘B’ → transform into a target data model (B1) → transmit to downstream (B2). Both workflows must be running on two different threads.

Appreciate someone can advice on how temporal help to implement above. Thanks.

2

Temporal is useful when processing relies on external APIs or can take a long time. From your description, it is not clear if your transformation needs these. If it is purely data manipulation, you could do them in a Kafka handler (which guarantees retries).