We have set up the temporal archival process to S3.
Question1:- When we set up the archival, we used to give a retention period for example let it be 10 days. So the workflows will be archived to S3 which is older than 10 days. Suppose if we decide to change the retention period to 5 days. Will archival process retrospectively archive workflows older than 5 days or will it apply only to the workflows which are going to execute going forward? What will happen to the workflows which ran between 6-10 days time period?
Question2:- Is there any way to manually force the archival process to run rather waiting for retention period?
Question1
Archival times should be calculated when workflow closes with the namespace retention set at that time. So changing retention does not have effect on already closed workflows (they will be archived at the already calculated time). Retention changes only apply to new closed workflows after retention is modified.
Question2
Not currently
Can we manually push the data into S3 Archival? will temporal archival can read it from temporal UI through archival option?
Can we manually push the data into S3 Archival?
I assume you could manually get workflow histories of completed executions and store them wherever needed. Is that the question?
will temporal archival can read it from temporal UI through archival option?
Yes, web ui should be able to display the archived visibility info in the “Archive” tab if archival is enabled. You can also get it via api for example using ListArchivedWorkflowExecutions
My question is, whenever temporal archives the workflow after retention we can query that using the Archival tab in temporal UI. If I create a workflow to push these workflow manually to the same S3 location ? Will temporal recognize these manually pushed data in S3 and Can i query it through Archival tab in Temporal UI? Can temporal allow Retention period 0 (zero)?
Yes, but note that the query functionality is different depending on the provider that you use, for s3 see visibility syntax here.
Can temporal allow Retention period 0 (zero)?
No, it’s min 1 day and max 30 days currently.
As per this documentation, i see this says we can have retention period as Zero. Check the section under “Test Setup”
./tctl --ns samples-namespace namespace register --gd false --history_archival_status enabled --retention 0
Good catch, thank you! There was another issue in that tctl command as well. Raised docs issue here.
We use S3 as our primary data store and the DB as a transactional one So we kept our Retention period as 1 day. The Visibility data will be archived immediately once the Workflow encounters the close event. But history archiver runs only after the retention period. Since Database is only transactional, there is chance we might change our Databases in our production. During such event, we want to run the history archiver. various options we are thinking. If you can add those functionalities it will be good for us. Please let us know the process to request one.
option 1. Please allow retention period as Zero.
option2- Force run the waiting closed workflow’s “DeleteWorkflowExecution” Event , so that it will start the history archiver process rather waiting for retention period.
option3 - Give us an option/feature to archive the history adhoc fashion.
if we manually try to archive it , then it will not be same as Temporal native integration archival. please suggest a solution for our problem.
Archival is currently limited and will be improved in the future.
There are plans to move both history and visibility archival to workflow execution close time (not retention time) which would fit your option 2, but there is not timeline for this yet that can be shared.
I will share your options with server team so it’s noted and will report back when there is something concrete in place.
Unfortunately right now, none of the 3 options you mentioned are possible with archival.
You can update your archival urls per namespace via tctl if that helps.
Thank you. Archiving the history as soon as the workflow execution close time will help us a lot. Is there any place/document we can check that these archival changes you mentioned are going to happen? Is there a way we can expedite it, We can ask our company to request one.
Feel free to add your use case to this issue, or create a new one with your specific requirements.
I think just upvoting archival issues in the Temporal repo and being vocal in the community would help drive fixes.