Hi, I’m doing some performance load using Temporal and workers written with go sdk.
Observing warnings in log:
{“level”:“warn”,“ts”:“2020-11-27T08:35:21.017Z”,“msg”:“Too many pending tasks”,“service”:“history”,“shard-id”:1,“address”:“172.17.0.2:7234”,“shard-item”:“0xc000466800”,“component”:“transfer-queue-processor”,“cluster-name”:“active”,“logging-call-at”:“queueAckMgr.go:184”}
How can I prevent to receiving these?
Thanks.
This indicates that DB or service is overloaded. What is your setup? Note that the service brought through the default docker-compose should be used for development only is not configured for any real load.
Yes, it was a default config, and one Cassandra instance.
Now I run with 3xCassandra cluster and use NUM_HISTORY_SHARDS=32 - warning not observed.
I see in documentation numHistoryShards: 512 - is it recommended value for production setup?
The numHistoryShards depends on the maximum size of the cluster you ever expect. The 512 is OK for a medium-size cluster. If you expect to grow to dozens of nodes then you can go higher. The maximum cluster we ran had 16k shards. Running too many shards for a small cluster would increase memory pressure on history nodes. Note that it is not possible to change the number of shards after the cluster is initialized.