I have a use case which requires looking up an existing workflow via search attribute – basically an existing workflow is signaled when an inbound REST API call is made, based on some external event. Ideally this call would simply contain the workflow ID but its an external call and I have no control over its content.
I have therefore implemented a workflow lookup via listExecutions and a query for various search attributes which I can obtain from the event.
This works fine in production, but in my integration test code which uses the Temporal internal test server, I am unable to call this API (see this forum post and this SDK issue).
My current test code has a terrible hack in it which injects the workflow lookup code, and in prod it does the usual listWorkflows, but in test it sets and uses the workflow id. This works but is messy and I’d like to avoid it.
Assigning low priority.
Test Server is intended for testing Workflow implementations, full on-par support of all administrative endpoints is not really a goal. Workflow logic shouldn’t trigger listopenworkflowexecutions, it’s an admin/operation endpoint.
If listing workflows is not intended to be used except for administrative purposes, how would I implement the use case I have described? Isn’t this a common scenario?
Note that listExecutions API has pretty tight rate limits as ES doesn’t scale in terms of the number of queries. If this scenario needs to scale, you have to use an external DB to perform the WorkflowID lookup.
Follow-up question… I don’t need this to scale so I was fine with searching for a workflow via search attributes. However, I am finding that I have a race condition: after my in-workflow call to upsertTypedSearchAttributes (Java SDK), the workflow is not immediately searchable by the upserted value.
Pulling in an external database dependency is of course possible but it feels like overkill for this use case.
I would have thought that finding existing workflows is such a common Temporal use case that this kind of logic would be supported out of the box without having to deal with this kind of race.
Unfortunately, it is impossible to provide read-after-write consistency as Elastic Search doesn’t support it.
Not quite true: the refresh parameter does support the true or wait_for values which do provide read-after-write consistency.
Specifying true would probably not be a good idea for load reasons, but it seems like a search attribute update could benefit from wait_for , which for active indices should only wait up to a second or so if ES is using its default configuration.
It might make sense to make the async nature of the current API explicit in the docs, and provide a new upsertTypedSearchAttributesSync API that specifies refresh=wait_for on the ES index call.