Our use case is app code that uses Kafka events emitted from a data system of ours that other business units interact with - these events define the start of the overall workflow and the custom search attributes values.
Currently for MVP we don’t seem to need a lot of “activities” - app code will handle Kafka events to start a workflow and then send “signals” (?) to pass down custom search attribute changes at any point. There may be a need for activities in some of the workflows in which we interact with other systems but at the moment not so much.
However I am a little confused whether this is this the right way to handle this? Send signals from app code to change status and custom attributes? But then how would child flows within a larger parent flow know when to finish - can workflows wait on a signal to consider itself “complete” and move to the next child workflow? or child workflow can wait on a custom search attribute value to determine if it can consider itself ‘done’ before moving to next child workflow??
edit - Im wondering if workflow condition is what I need…
It is hard to give a concrete recommendation without knowing your use case.
It is not clear why you want to use Temporal instead of just a DB table to store “search attributes.”
Workflows that listen to signals (which are created from Kafka messages) is a widespread pattern. Such workflows can decide to exit based on their business logic at any time. It is common to use SignalWithStart to deliver signals as messages can come out of order.
Its for a simple (for now) workflow service at our business. Simply put - get a kafka event on the backend service, send off a Temporal signal (?) which the workflow then awaits a condition to say “yes this thing is done, now move to next child workflow”. There wont be many activities as such yet - maybe some of the child workflows will require an activity to do something like create a Slack channel or something. Overall trying to determine how to use Kafka to run most things and Temporal to just act on something sent from the backend (via Signals or whatever) in order to represent the state of a particular order running through the workflow. I need to look at the SignalWithStart - I just seen this now on the typescript samples. So it takes the “signal” and “signalArgs” as options. I will need to look at the “mutex” TS sample project a bit more - its using the Temporal client from within the activities but can see the `signalWithStart! usage
Thought to use search attributes as a simple way to store what is currently just a small set of simple strings that we can display on a frontend - may spin up a DB at some point in the service but trying to avoid right now. We can avoid doing this and also just do data lookups from data sources too if its deemed to be a bad idea to add to search attributes.
You mentioned child workflows. Why do you need them instead of running activities directly from a workflow that received the signal?
Why do you update the search attributes? Why don’t you store data inside the workflow and query the workflow directly? Search attributes are not designed as per workflow storage, and they are used only for indexing workflows.
So child workflows, i.e. executeChild which we are using - we don’t need them as such I guess - it seemed like a logical grouping off all sub-workflows within a larger workflow? Or is that not the right thought process? We could just run directly without wrapping in executeChild (Ive forgotten what executeChild gives as such - I know it seems to show things as separate flows in the Temporal UI). So in terms of code, having one top level workflow function with a bunch of signal handler conditions would be ok?
I appreciate the search attributes are for indexing - so it seems like we shouldn’t store here for these reasons. When you say “store in the workflow and query the workflow directly” - what do you mean exactly - like local state in a workflow definition? If so, what happens during Temporal cluster crashes for example - would this be persisted in Temporal cluster DB layer ?
Children are the fine way to separate pieces of functionality. But looking at this comment:
There wont be many activities as such yet - maybe some of the child workflows will require an activity to do something like create a Slack channel or something.
I don’t think you need a separate workflow to run a single activity.
what do you mean exactly - like local state in a workflow definition? If so, what happens during Temporal cluster crashes for example - would this be persisted in Temporal cluster DB layer ?
Yes, all workflow variables are fully durable in Temporal. So no need to save them in any external DB explicitly.
Yeah fair point - don’t need abstractions for single activities.
Workflow variables being durable - I think I missed this point initially and its still a little mind bending if it works how I think. Now it makes sense why I saw many of the query examples querying local state within a workflow…
But I do have questions:
1 - the data stored inside a workflow; this would literally be assigned to a local variable and then returned via a query?
2 - I’m guessing this variable within a workflow is associated with the particular workflow-id - its not getting overwritten?
3 - is there a limit to what can be stored? There shouldn’t be a lot of data
4 - Is there a case for when this isn’t recommended and using an activity to retrieve data from elsewhere OR store the data sent in a signal to a DB is better?
1 - yes
2 - workflow state is per workflow id.
3 - the limit is memory allocated to the worker process. The practical limit is usually the size of workflow history and individual payloads of signals, activity results, etc.
4 - DB is for storing pure data (user name, address, etc.) or historical records (for example the historical order information). It is not for storing the current state of the workflow. For example the current state of an order being fulfilled.