Temporal s3 archival issues

Community Support
1

i see this log frequently , my s3 configuration seeems correct and i also see data in bucket. not sure whats going wrong

{“level”:“error”,“ts”:“2021-02-02T04:13:20.707Z”,“msg”:“failed to archive history, will move on to deleting history without archiving”,“service”:“worker”,“component”:“archiver”,“wf-namespace”:“temporal-system”,“wf-id”:“temporal-archival-783”,“wf-run-id”:“0591ac69-d729-4732-9752-1dab9500f7ed”,“wf-task-queue-name”:“temporal-archival-tq”,“wf-type”:“archivalWorkflow”,“shard-id”:382,“archival-request-namespace-id”:“35138b5a-8215-469d-b68a-482b0b7fb271”,“archival-request-namespace”:“mynamespace”,“archival-request-workflow-id”:“environment-summary-report”,“archival-request-run-id”:“ad203df5-b2c2-4fc3-afab-b6abf61211b9”,“archival-request-branch-token”:"[10 . . . . .]",“archival-request-next-event-id”:13,“archival-request-close-failover-version”:0,“archival-URI”:“s3://mytemporalacrivebucket”,“error”:“activity error (type: uploadHistoryActivity, scheduledEventID: 1291, startedEventID: 1292, identity: 1@my-temporal-worker-7fd76f5644-z2tv9@): upload non-retryable error”,“logging-call-at”:“handler.go:148”,“stacktrace”:“go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/log/loggerimpl.(*replayLogger).Error\n\t/temporal/common/log/loggerimpl/replay.go:87\ngo.temporal.io/server/service/worker/archiver.(*handler).handleHistoryRequest\n\t/temporal/service/worker/archiver/handler.go:148\ngo.temporal.io/server/service/worker/archiver.(*handler).handleRequest.func1\n\t/temporal/service/worker/archiver/handler.go:115\ngo.temporal.io/sdk/internal.(*dispatcherImpl).NewCoroutine.func1\n\t/go/pkg/mod/go.temporal.io/sdk@v1.3.0/internal/internal_workflow.go:920”}

2

NOTE: archival is an experimental feature.

the above error means archival (after few retries) fails. It can be the case that the error is timeout error.

One more thing, the error message is also correct, if the archival (after a few reties) fails, the workflow history will be deleted (this is not desired behavior, and the feature is still experimental). Please create an issue if archival is an onboarding blocker.

3

Thanks @Wenquan_Xing do you mean entrie archival is experimental or the s3 archival is experimental?

4

right now the archival feature is experimental

btw, we are evaluating the path forward for archival

5

@madhu If you have specific requirements related to archival let us know in this thread.

6

I don’t have a specific requirement @maxim,i felt it will be awesome
a)t if we can get the = compare only limitation removed from s3 .
b) and if we can have guaranteed eventual archival of completed workflows.

These are not blocker for us but certainly there in our nice to have list.

1 Like
7

@maxim Is the behaviour where the workflow will be deleted, even though archival fails, still the same?

Also, what is is the timeline the Archival feature to remove the experimental tag?

8

Is the behaviour where the workflow will be deleted, even though archival fails, still the same?

Yes this edge case can still happen.

what is is the timeline the Archival feature to remove the experimental tag?

There is not a timeline for this yet for OSS.

9

its been a while, now the archival is becoming a soar wound in our production systems. any plans to have a full fledged archival implementation? and remove the experimental feature.

my requirements are still the same

archival should be guranteed.
= comparision on s3 to be addressed.

additioanlly if we can have make file system based archival production grade nothing like it.
most azure,aws uses can still use this backed by EFS or azure file store.

10

Archival is durable since this change: Enable durable archival by MichaelSnowden · Pull Request #3828 · temporalio/temporal · GitHub

11

thanks nice to know that. so its durable by default, i dont have to do anything. if so that leaves us with only problem that the archival page in ui is just unusable.