Retrieving archived workflow data from s3 via UI and tctl workflow list archived

Community Support
, , , ,
1

I’m facing issues with setting up Archival with s3, though i’m half way there.

user@TM00102 ~ % tctl -ns benchtest n desc
Name: benchtest
Id: 7029315a-7234-4b99-873d-2ad64363a9b7
Description: 
OwnerEmail: 
NamespaceData: map[string]string(nil)
State: Registered
Retention: 1h0m0s
ActiveClusterName: active
Clusters: active
HistoryArchivalState: Enabled
IsGlobalNamespace: false
FailoverVersion: 0
FailoverHistory: []
HistoryArchivalURI: s3://dev-test-bucket-123
VisibilityArchivalState: Enabled
VisibilityArchivalURI: s3://dev-test-bucket-123
Bad binaries to reset:
+-----------------+----------+------------+--------+
| BINARY CHECKSUM | OPERATOR | START TIME | REASON |
+-----------------+----------+------------+--------+
+-----------------+----------+------------+--------+

Archival is enabled for history and Visibility, We are able to view the list of archived workflows in web UI and in the s3 - we are able to see the actual data in the bucket with key as /<namespace_id>/<Worklow_id>/<run_id>/0/0.
Content in the file ‘0’ is not encrypted and i’m able to read it, only payload data is encoded.
I’m also able to view the same data using tctl workflow show --Worklow_id <Worklow_id> --run_id <run_id> for archived workflow as well.

The problem is
(1) We are getting error " 404 This is not the Workflow you are looking for" when trying to access the history data of archived workflow in UI.

image
image3146×956 225 KB

the tab and URL is moved from /archival to /workflow/<Workflow_id>/<run_id> upon selecting a workflow from archived workflows.
image
image2602×1472 180 KB

(2) I’m not able to list archived workflow using “tctl workflow listarchived” command.

''tctl workflow listarchived                                                                                                           
Error: Option query is required""

What is the query we have pass here, i see ‘SQL like query’ in help. can someone share some examples for this.

(3) Major confusion is Workflow data is present in S3, list of workflows are accessible via UI but not their history data, workflow show works for archived data while listarchived is not. How do we debug if archival is working, is it using “workflow show or workflow listarchived” command ?

can someone sort this out, i see few others are facing this issue and they are using older version because of this.

i’m using 1.25.2 for server and 2.31.2 for UI.

2

What is the query we have pass here, i see ‘SQL like query’ in help. can someone share some examples for this.

You can see visibility query syntax for s3 archiver here if that helps.

Im not sure about the UI issue and showing history data, which web ui version are you deploying?

How do we debug if archival is working, is it using “workflow show or workflow listarchived” command ?

Would use listarchived to test

1 Like
3

Thanks for the reply @tihomir, tctl workflow listarchived worked as per the doc.
command for above shared example :

user@TM00102 ~ % tctl -ns benchtest workflow listarchived -q "WorkflowId='HelloActivityWorkflow'" --print_datetime
   WORKFLOW TYPE   |      WORKFLOW ID      |                RUN ID                | TASK QUEUE |      START TIME      |    EXECUTION TIME    |       END TIME        
  GreetingWorkflow | HelloActivityWorkflow | ef4f311b-0a81-4d88-b291-b81b32b04762 |            | 2024-09-06T05:46:05Z | 2024-09-06T05:46:05Z | 2024-09-06T05:46:07Z  
  GreetingWorkflow | HelloActivityWorkflow | 5858430c-1904-4e77-aee7-1752b74f0c22 |            | 2024-09-06T08:51:28Z | 2024-09-06T08:51:28Z | 2024-09-06T08:51:30Z  
  GreetingWorkflow | HelloActivityWorkflow | 320608c4-fd4b-4693-910c-4642cd3da467 |            | 2024-09-06T12:24:33Z | 2024-09-06T12:24:33Z | 2024-09-06T12:24:33Z  
  GreetingWorkflow | HelloActivityWorkflow | 2067b584-44b3-41fb-a3a3-d9883c1a04c4 |            | 2024-09-06T12:38:00Z | 2024-09-06T12:38:00Z | 2024-09-06T12:38:01Z  
  GreetingWorkflow | HelloActivityWorkflow | 81df865c-23ea-4d75-ace0-693d2f134a9b |            | 2024-10-28T06:21:47Z | 2024-10-28T06:21:47Z | 2024-10-28T06:21:48Z

We have tested archival with a few different UI versions from 2.9 to 2.31.2 and 2.14, 2.16, 2.27 in between, retrieving archived workflow history from UI never worked for us.
I am not sure if this is a config issue or an bug that is already posted in github issues.

4

Also my guess is, Visibility archival feature was fixed in 2.14.0 and retrieval of history data from s3 of an archived workflow was not. Referring to this thread.