Context deadline exceeded issue

We are still getting the context deadline exceeded error while triggering a new workflow.

We are deploying the temporal server with helm commands.

We do have a working system in another cluster with following versions.

Azure (Old and not been upgraded but Working without any issues) :
Temporal version :: 1.8.1
Kubernetes version :: v1.20.13

GCP (Working)
Temporal version :: 1.14
Kubernetes version :: v1.22.10-gke.600

GCP (Not Working) Recently created the cluster
Temporal version :: 1.16.2
Kubernetes version :: v1.22.9-gke.1500

Our Goversion was 1.12 and we updated the code to 1.16 and temporal sdk to 1.16.0 … even after that we are having same issue .

I have also attached the history/frontend pod logs or your Reference in the Google Drive
Temporal - Google Drive

Please go through the logs and let us know to mitigate the issue.