Greetings. Hoping to advice for a workflow that quickly emits 10,000 activity tasks. Each task represents an S3 multi-part (up to 10,000). I’m using temporal with docker-compose and assigned 16Gb. The tasks are emitted calling ResponseWorkflowTaskCompleted, but, the call to the server exceeds timeout (the default of 10 seconds). I don’t think it’s because the message-size is exceeded, there is nothing in the logs to indicate that.
Yes, I could recode to call child-workflows to reduce the activity count, but it just adds complexity.
My question is two fold. First, I tried increasing the RpcTimeout to 60 seconds, but it appears to make no difference, it still uses 10 seconds (yes, I checked in the debugger to ensure it’s being set in GrpcDeadlineInterceptor) – any ideas (I still need to debug deeper)? Second, what configuration changes would be needed to speed up the server accepting the message? What’s reasonable – should it even accept 10,000 tasks in < 10s? For example, more Cassandra instances/partitions/whatever? Is there any documentation on how to size the installation, what metrics to track, etc.?