Hello. I have some temporal workflows running on a kubernetes pod. One of the workflows starts an activity, which is doing a bunch of BatchWriteItem calls to DynamoDB(close to 40k requests by an activity execution). A snipper of the activity BatchWriteItem logic code is below:
av, err := attributevalue.MarshalMap(msg)
if err != nil {
return err
}
writeRequests = append(writeRequests, types.WriteRequest{
PutRequest: &types.PutRequest{
Item: av,
},
})
if len(writeRequests) == maxBatchSize {
input := &dynamodb.BatchWriteItemInput{
RequestItems: map[string][]types.WriteRequest{
repo.tableName: writeRequests,
},
}
out, err := repo.store.BatchWriteItem(ctx, input)
if err != nil {
return err
}
}
When executing a workflow, this activity runs for around 10 minutes without any issues, and suddenly I see this error log in the pod
2025/03/17 00:02:40 INFO Task processing failed with error Namespace <namespace> TaskQueue <queue-name> WorkerID 1@kubernetes-worker-pod-name@ WorkerType ActivityWorker Error context deadline exceeded
I set the Temporal Activity StartToClose timeout to 2 hours, so that is not the reason for error here. I checked the temporal worker pod CPU and memory usage and they are within limits. I also noticed there are no throttles write requests in DynamoDB dashboard. Not sure what else is causing this issue. Can I get some pointers on the cause of this error?