Temporal Archival retrieval

Community Support
1

I was going through the document regarding the archival process. We are planning to integrate our archival to S3. I have couple of questions.

  1. What set of attributes will be archived to S3? can this include the custom search attributes as well?
  2. Since Temporal has the S3 native integration. If we are querying/searching a workflow using workflowid/ Run id, if that workflow detail is archived to S3, will Temporal able to bring the details of the workflow from S3 similar to bringing details using custom search attributes from Elastic Search? or we need to code for archival retrieval process separately.
2

The archived data row is described here: temporal/message.proto at master · temporalio/temporal · GitHub. Custom search attributes are included in search_attributes field.

If I understand code correctly indexes are described here: temporal/visibilityArchiver.go at master · temporalio/temporal · GitHub.

Please note, archival feature is in “experimental” stage. I would even call it “early experimental”. We haven’t test it for a while and don’t have any test coverage besides unit and integration(?) tests. Some work are planed there but I can’t give you a time frame for now.

3

Just to add, this thread is a bit older but has good information to question 1 (what data is stored during the archival process)

4

Alex, Thank you.
Qn1. In temporal UI we have the option to see the Archival. My understanding this comes from S3. correct me if I am wrong.

image
image796×222 9.02 KB

Qn2. Regarding the data retrieval from S3, this is the what mentioned in the document

Retrieve archives#

You can retrieve archived Event Histories by copying the workflowId and runId of the completed Workflow from the log output and running:

***./temporal --ns samples-namespace wf show --wid <workflowId> --rid <runId>***

Here we are just passing the workflowid and runid. Its more generic we are not specifying the store ,so my understanding is that the temporal will internally retrieves the information based on the workflowid either from (DB/ES) or from archival (s3).

Qn3. Can Query method handles the s3 retrieval?

5

Qn4 . When the data is archived to s3 from both database and Elastic Search (including our custom search attributes). Will it be stored in one object or two separate object?

Qn5 . The data in Database is encrypted , will the archived S3 will have the same encryption? Likewise The Elastic Search data is not encrypted will the S3 also not encrypted?