Hi Team, currently we are trying to publish notifications for history events of workflow execution as it progresses.
We explored GetWorkflowExecutionHistory but it sends all the events from the start and not just recent updates, so we have to track till what point we have send notifications and send only remaining.
We also explored using PollWorkflowTaskQueue which returns set of history events it’s processing but when worker is crashed/restarted, it is pulling entire history again which is creating duplicate events for our notifications.
To solve above 2 issues, we cannot create map and track last published events(in-memory) as when our service pod restarts then we will lose that data and if we have to store it in persistence then we will have additional latency for the same. Any recommendations around how we can get new set of realtime history events for a workflow execution?
You could look how tctl does this for its observe
command, it does keep track of last event but think thats the approach you will need to do in the end:
Links here, and here
.
Thanks @tihomir for the response. Yes, we have explored this also but as we need to pass nextPageToken to get new events as part of GetWorkflowHistory and in case our service(which has the logic of looping on GetWorkflowHistory) goes down then we will not have nextPageToken so that leads again in storing nextPageToken in persistance. We are looking on how we can skip persisance like can we store nextPageToken in workflow/activity side any way which can retrieved even if service goes down so that we can run GetWorkflowHistory loop as part of activity?
I don’t understand your requirements. How do you know which events are new since the service crash if you don’t have the page token or at least eventId?
@maxim Thanks for the response. Yeah makes sense we will try to use eventId or nextPageToken to be stored. When we call GetWorkflowHistory, can you confirm if it always comes in ascending sequence of eventId?
if it always comes in ascending sequence of eventId
Yes, its always in same order, the number of events that are returned in single page depends on dynamic config frontend.historyMaxPageSize