This is not quite a support topic. I just wanted to bounce some ideas around, and figured others may have similar questions or ideas.
To search / filter workflows, the current recommendation if I understand correctly is to use a Temporal set-up with Kafka, and Elasticsearch.
This makes sense if you are working with high traffic applications considering both Kafka, and Elasticsearch were built for scale.
Now, for simpler use-cases, and/or lower traffic applications, this feels a bit overkill. Both Kafka, and Elasticsearch are not readily available in all cloud providers, and they can be quite costly to run (with or without a managed service provider).
There seems to be a few longer term options:
- Possibly removing Kafka (https://github.com/temporalio/temporal/pull/295)
- Support searching / filtering (search attributes) with the SQL persistence as well (Postgresql can handle this quite well)
- Similar to the previous point, supporting more search engines (e.g. MeiliSearch)
And for the short-term, it seems like the suggestion from what I have seen is to essentially index the workflow through activities. i.e. Making it a part of your domain or building a dedicated service to capture / index workflows.
Arguably, baking workflows into a specific domain however, does not always feel like the right solution. Workflows can arguably cross various domain boundaries, and I feel the beauty of it is that you can focus on the overall / high-level business logic (orchestration logic).
And building a dedicated service to capture workflows feels like boilerplate which a workflow engine like Temporal aims to reduce.
I’d be interested in knowing what y’all think, and how others are approaching this.