Do you have both SDK and server metrics set up? It would help figure out latencies, especially on your worker side, see here for more info.
- name: NUM_HISTORY_SHARDS
value: “512”
This seems to use the default 512 and is often too low, see here for general recommendations.