Temporal s3 archival issues

#1

i see this log frequently , my s3 configuration seeems correct and i also see data in bucket. not sure whats going wrong

{“level”:“error”,“ts”:“2021-02-02T04:13:20.707Z”,“msg”:“failed to archive history, will move on to deleting history without archiving”,“service”:“worker”,“component”:“archiver”,“wf-namespace”:“temporal-system”,“wf-id”:“temporal-archival-783”,“wf-run-id”:“0591ac69-d729-4732-9752-1dab9500f7ed”,“wf-task-queue-name”:“temporal-archival-tq”,“wf-type”:“archivalWorkflow”,“shard-id”:382,“archival-request-namespace-id”:“35138b5a-8215-469d-b68a-482b0b7fb271”,“archival-request-namespace”:“mynamespace”,“archival-request-workflow-id”:“environment-summary-report”,“archival-request-run-id”:“ad203df5-b2c2-4fc3-afab-b6abf61211b9”,“archival-request-branch-token”:"[10 . . . . .]",“archival-request-next-event-id”:13,“archival-request-close-failover-version”:0,“archival-URI”:“s3://mytemporalacrivebucket”,“error”:“activity error (type: uploadHistoryActivity, scheduledEventID: 1291, startedEventID: 1292, identity: 1@my-temporal-worker-7fd76f5644-z2tv9@): upload non-retryable error”,“logging-call-at”:“handler.go:148”,“stacktrace”:“go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/log/loggerimpl.(*replayLogger).Error\n\t/temporal/common/log/loggerimpl/replay.go:87\ngo.temporal.io/server/service/worker/archiver.(*handler).handleHistoryRequest\n\t/temporal/service/worker/archiver/handler.go:148\ngo.temporal.io/server/service/worker/archiver.(*handler).handleRequest.func1\n\t/temporal/service/worker/archiver/handler.go:115\ngo.temporal.io/sdk/internal.(*dispatcherImpl).NewCoroutine.func1\n\t/go/pkg/mod/go.temporal.io/sdk@v1.3.0/internal/internal_workflow.go:920”}

#2

NOTE: archival is an experimental feature.

the above error means archival (after few retries) fails. It can be the case that the error is timeout error.

One more thing, the error message is also correct, if the archival (after a few reties) fails, the workflow history will be deleted (this is not desired behavior, and the feature is still experimental). Please create an issue if archival is an onboarding blocker.

#3

Thanks @Wenquan_Xing do you mean entrie archival is experimental or the s3 archival is experimental?

#4

right now the archival feature is experimental

btw, we are evaluating the path forward for archival

#5

@madhu If you have specific requirements related to archival let us know in this thread.

#6

I don’t have a specific requirement @maxim,i felt it will be awesome
a)t if we can get the = compare only limitation removed from s3 .
b) and if we can have guaranteed eventual archival of completed workflows.

These are not blocker for us but certainly there in our nice to have list.

1 Like