I need to implement pagination with sorting by start/end time, status and filtering by status and workflow id prefix for workflow executions.
I found, that I can implement filtering using query parameter of ListWorkflowExecutions.
Also, I found that there was possible to have ORDER BY clause in the ListWorkflowExecutions (@tihomir Mentioned it here 2 years ago)
But now (1.24.2.1) adding ORDER BY and LIMIT clauses are not supported (source code)
My question is: is there any way to implement pagination with sorting and filtering using out of the box functionality?
Also, I noticed, that temporal web provides frontend based pagination (load all and split to pages on UI). This won’t work for my case
P.S.: I am really struggling when try to understand capabilities of API.
Can’t find single source of truth about the API and answers for simple questions like “what is the default sorting for ListWorkflowExecutions?”. Have to dig forums, examples, documentation for different languages and cli, and even source code of the temporal.
It would be really helpful to have something similar to OAS spec for Temporal APIs
Also, I found that there was possible to have ORDER BY clause in the ListWorkflowExecutions (@tihomir Mentioned it here 2 years ago)
But now (1.24.2.1) adding ORDER BY and LIMIT clauses are not supported (source code)
correct, it was disabled by default as custom ordering by is pretty heavy operation. you can still enabled it via dynamic config
system.visibilityDisableOrderByClause
(set it to false)
but its recommended to use mentioned default sort order instead if possible
I need to implement pagination with sorting by start/end time, status and filtering by status and workflow id prefix for workflow executions.
not sure whats the issue here, you can filter by StartTime / CloseTime, ExecutionStatus and WorkflowId STARTS_WITH works too
do you have custom rules for pagination? default page size is 1K and you can change that via dynamic config frontend.visibilityMaxPageSize
if you want to cap it to something smaller. most visibility apis also allow you to specify smaller page size than the cap you define in dynamic config
The issue is that temporal API doesn’t support limit/offset.
This mean, if I need to implement 30 rows per page pagination and user wants to access page â„–15:
if frontend.visibilityMaxPageSize = 1000
need to make page request, retrieve 970 unneeded rows of data, filter them out, return result to user
if frontend.visibilityMaxPageSize = 30
need to make 15 page request (because next token is required to retrieve next page).
The SDK provides a pageSize and a nextPageToken, so cursor-based pagination, which makes jumping to a specific page difficult / inefficient.
I’ll have to change the UI to match that pagination style. I hope this feature can be added.
As a side note...
I’m currently spending my time refactoring the codebase away from having records in my own DB that represent the workflow (basically an “Aggregate representing the business process”, which I could query and filter fully using SQL) to using the Temporal SDK to fetch data for the UI directly from the workflow state.
Temporal has been challenging and nuanced to build on tbh, even though it has some fantastic features. Trying to keep this local state in-sync with the workflow stat, trying to start workflows transactionally with this record creation, and needing workflows to trigger other workflows is difficult with these process boundaries in the way.
Adding offset-based pagination may seem like an unimportant feature, but Temporal seems to be quite all-or-nothing. The docs recommend you treat the workflow as a service, give examples of modelling a shopping cart as a workflow (for example), so if you want to use it as a service, we need basic functionality like this and an easier way to list/paginate workflows along with their current state or query in one round trip (ideally, to avoid N+1 queries).
I’m having to hack the life out of custom search attributes to render the table of workflows with the columns I want to filter on and display (considering the number of custom search attributes is limited per namespace to quite a low number). It’s pretty awkward that the custom search attributes are all Protobuf based objects too, atleast with the connection.workflowService.listWorkflowExecutions() function (the temporalClient.workflow.list() function doesn’t even accept the nextPageToken).
I hope this feedback reaches the dev team. These features might not be as important if the workflows are hidden in the background, but if the business process requires internal UI to support them, they become much more of a front and centre thing to work with!
The difficulty I had with maintaining an external DB was trying to filter workflows, e.g. by something app-specific like Company Name and workflow state (running, complete, waiting for a signal, etc.) at the same time. You have to get that workflow state into the DB too, (otherwise you’re trying to query two different data stores and produce a sensible page of results from it) then keep it in-sync.
If the workflows had a way to sync state to an external DB /read model, e.g. like a hook that gets called if someone in the Temporal UI or CLI reset/restarted a workflow, that would be useful. Otherwise, I guess you’d need to insert an activity in between every real activity to sync the state.
The thing I’ve found most difficult about this is the process boundary or probs better team/workflow boundary. You either create that record in the external DB first and then trigger the workflow (which isn’t transactional without extra work) and it also makes running child-workflows infeasible this way, or you trigger the workflow and then create the record using an activity which ends up with a CQRS-style problem of rendering the UI. I’ve been interested in Nexus, I hope it might straighten this challenge out by making there be only one way to have workflows trigger other workflows across process/team boundaries and be able to track them both in the UI
