Take a look at this post for info on more server and sdk metrics that can help tune your workers (worker tuning guide in docs), as well as here and here for recommendations for load testing/ prod setup.
From the described latencies it seems the bottleneck is related to worker capacity.