Task processing failed with error .... WorkerType ActivityWorker Error context deadline exceeded

Community Support
1

Hello. I have some temporal workflows running on a kubernetes pod. One of the workflows starts an activity, which is doing a bunch of BatchWriteItem calls to DynamoDB(close to 40k requests by an activity execution). A snipper of the activity BatchWriteItem logic code is below:

                av, err := attributevalue.MarshalMap(msg)
		if err != nil {
			return err
		}

		writeRequests = append(writeRequests, types.WriteRequest{
			PutRequest: &types.PutRequest{
				Item: av,
			},
		})

		if len(writeRequests) == maxBatchSize {
			input := &dynamodb.BatchWriteItemInput{
		RequestItems: map[string][]types.WriteRequest{
			repo.tableName: writeRequests,
		        },
	         }
                out, err := repo.store.BatchWriteItem(ctx, input)
		if err != nil {
			return err
		}
	}

When executing a workflow, this activity runs for around 10 minutes without any issues, and suddenly I see this error log in the pod

2025/03/17 00:02:40 INFO Task processing failed with error Namespace <namespace> TaskQueue <queue-name> WorkerID 1@kubernetes-worker-pod-name@ WorkerType ActivityWorker Error context deadline exceeded

I set the Temporal Activity StartToClose timeout to 2 hours, so that is not the reason for error here. I checked the temporal worker pod CPU and memory usage and they are within limits. I also noticed there are no throttles write requests in DynamoDB dashboard. Not sure what else is causing this issue. Can I get some pointers on the cause of this error?