Hi. So, I have a bunch of services that communicate via kafka topics and currently they form an event chain (i.e. service A produces to a topic, which is consumed by service B, it produces to another topic which is consumed by service C and so on. It is slightly more complicated in reality, but that is not relevant).
I need to change this to an orchestrated flow, but I couldn’t find a way to do that easily with temporal, because workflow would need to react to kafka messages and I’m not sure how to do that properly.
I saw a suggestion to have workflow wait for signals and have a separate service that would listen to kafka topics and signal workflow based on the trigger. Though with this approach I would have to implement a complex retry process in case the signalling fails and also it splits the workflow logic into two(one in temporal and another in this separate service)
Can you clarify what you mean by “signalling fails”? If the Temporal server accepts the signal, it will be provided to the workflow (if the workflow chooses not to handle it is a different matter). One might expect you wouldn’t commit a Kafka offset of a message that the Temporal server didn’t accept a signal for. Depending on system details, you may have to have the workflow be smart enough to ignore duplicate signals.
Can you clarify this a bit? Are you saying it’s split because there is an out-of-band process transferring Kafka messages to Temporal signals? Such a process has to exist somewhere.
Networking issues, infrastructure issues. Something is not working, so signal is not coming through.
I for sure would not commit an offset in that case, but there would be a need to implement some process that would report on that happening, possibly limit the number of retries, etc.
I’m fine with that, if this is a proper way to go with handling kafka/temporal interactions.
So, with this approach I will have a workflow that waits for signals and start activities in a specified order.
And another service that sends signals based on kafka messages.
If I need to change the order for some reason - I will have to change both at the same time.
Yes, I am afraid that these are the burdens of tunneling messages from one system to another. Once the signal is accepted by the Temporal server, it will always be available for processing if the workflow has a handler for it (IIRC there is no failure scenario where the server accepts it but it does not get recorded in history for processing). Signals are handled in the order signalled.
Consider removing Kafka completely for interprocess communication. Temporal activities already use task queues that are internal to the Temporal service. So you can host different sets of activities in different services. This way a lot of complexity is eliminated.