Weird Behaviour with Archival - Requires HttpGet before archiving

I need to add the below code in the archival code. (first line of Archvie method in VisibilityArchiver.go & HistoryArchiver.go under s3store package)
If not the archival doesn’t work
Not sure why.

http.Get("MY-S3-URL")

Temporal is running on kubernetes.

I had a hard time gettting archival to work,
I added the above code to check if the egress is setup properly, and right then the archival started to work.

Not sure what http.Get("MY-S3-URL") does to get the archival to work magically.

FYI - S3 I’m using is a flavour of AWS S3

Could it be that you are using presigned s3 urls? Uploading objects using presigned URLs - Amazon Simple Storage Service

I don’t believe that’s the case.

I have listed down the errors I see.

In most cases the error is : Context deadline exceeded.

Want to understand what causes this error?

[
  {
    "level": "error",
    "ts": "2022-01-12T09:20:19.346Z",
    "msg": "Archive method encountered an non-retryable error.",
    "service": "history",
    "archival-request-workflow-id": "s-tst-1",
    "archival-request-workflow-type": "GreetingWorkflow",
    "archival-request-close-timestamp": "2022-01-12T09:20:17.453Z",
    "archival-request-status": "Completed",
    "archival-URI": "s3://s3-archival",
    "archival-archive-fail-reason": "failed to write history to s3",
    "error": "RequestCanceled: request context canceled\ncaused by: context deadline exceeded",
    "logging-call-at": "visibilityArchiver.go:122",
    "stacktrace": "go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/archiver/s3store.(*visibilityArchiver).Archive.func1\n\t/temporal/common/archiver/s3store/visibilityArchiver.go:122\ngo.temporal.io/server/common/archiver/s3store.(*visibilityArchiver).Archive\n\t/temporal/common/archiver/s3store/visibilityArchiver.go:151\ngo.temporal.io/server/service/worker/archiver.(*client).archiveVisibilityInline\n\t/temporal/service/worker/archiver/client.go:260"
  },
  {
    "level": "error",
    "ts": "2022-01-12T09:27:00.163Z",
    "msg": "failed to send signal to archival system workflow",
    "service": "history",
    "shard-id": 3,
    "address": "100.127.29.246:7234",
    "shard-item": "0xc000111100",
    "archival-caller-service-name": "history",
    "archival-archive-attempted-inline": true,
    "archival-request-workflow-id": "s-tst-2",
    "wf-id": "temporal-archival-818",
    "error": "context deadline exceeded",
    "logging-call-at": "client.go:297",
    "stacktrace": "go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/worker/archiver.(*client).sendArchiveSignal\n\t/temporal/service/worker/archiver/client.go:297\ngo.temporal.io/server/service/worker/archiver.(*client).Archive\n\t/temporal/service/worker/archiver/client.go:190\ngo.temporal.io/server/service/history.(*transferQueueTaskExecutorBase).recordWorkflowClosed\n\t/temporal/service/history/transferQueueTaskExecutorBase.go:196\ngo.temporal.io/server/service/history.(*transferQueueActiveTaskExecutor).processCloseExecution\n\t/temporal/service/history/transferQueueActiveTaskExecutor.go:313\ngo.temporal.io/server/service/history.(*transferQueueActiveTaskExecutor).execute\n\t/temporal/service/history/transferQueueActiveTaskExecutor.go:116\ngo.temporal.io/server/service/history.(*transferQueueActiveProcessorImpl).process\n\t/temporal/service/history/transferQueueActiveProcessor.go:331\ngo.temporal.io/server/service/history.(*taskProcessor).processTaskOnce\n\t/temporal/service/history/taskProcessor.go:269\ngo.temporal.io/server/service/history.(*taskProcessor).processTaskAndAck.func1\n\t/temporal/service/history/taskProcessor.go:221\ngo.temporal.io/server/common/backoff.Retry\n\t/temporal/common/backoff/retry.go:103\ngo.temporal.io/server/service/history.(*taskProcessor).processTaskAndAck\n\t/temporal/service/history/taskProcessor.go:248\ngo.temporal.io/server/service/history.(*taskProcessor).taskWorker\n\t/temporal/service/history/taskProcessor.go:171"
  },
  {
    "level": "info",
    "ts": "2022-01-12T07:17:50.855Z",
    "msg": "failed to perform visibility archival inline",
    "service": "history",
    "shard-id": 4,
    "address": "100.127.40.255:7234",
    "shard-item": "0xc00058f900",
    "archival-caller-service-name": "history",
    "archival-archive-attempted-inline": true,
    "archival-request-workflow-id": "s3-test-10",
    "archival-URI": "s3://s3-archival",
    "error": "RequestCanceled: request context canceled\ncaused by: context deadline exceeded",
    "logging-call-at": "client.go:237"
  }
]