Getting `signed by unknown authority` error with certificates generated from tls-simple repo

Community Support Server Deployment
,
1

I have setup a self-hosted temporal in GKE using the helm charts with cloud-sql and postgres which is working. Now I am trying to add TLS to this. My goal is to have temporal server check and verify the certificates the client presents (go sdk) during connection. I have tried to use the certificates generated from tls-simple but keep getting the signed by unknown authority error. Here are the steps I followed:

  1. edit cluster.conf to include localhost as DNS SAN:
[req]
default_bits = 4096
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[dn]
C = US
ST = WA
O = Test Cluster, Inc.
CN = tls-sample
[req_ext]
subjectAltName = @alt_names
[alt_names]
DNS.1 = tls-sample
DNS.2 = localhost
IP.1 = ::1
  1. Ran generate-test-certs.sh script in the tls-simple directory.
  2. created secreted with the ca.cert, cluster.key, cluster.pem and mounted them to the deployments.
  3. Setup the tls config as:
tls:
       frontend:
         server:
           certFile: /etc/certs/cluster.pem
           keyFile: /etc/certs/cluster.key
           requireClientAuth: true
           clientCaFiles:
             - /etc/certs/ca.cert
         client:
           serverName: tls-sample
  1. Port-forwarded the frontend-headless service and tried to connect my go-client. Got the failed to verify certificate: x509: certificate signed by unknown authority error.
    Go code:
//The client is a heavyweight object that should be created once per process.
	cert, err := tls.LoadX509KeyPair(
		"/home/rijalbasanta123/Downloads/temporal-base-code (1)/hello-world/certs/client.pem",
		"/home/rijalbasanta123/Downloads/temporal-base-code (1)/hello-world/certs/client.key",
	)
	if err != nil {
		log.Fatalln("Unable to load certs", err)
	}
	// Create the client object just once per process
	c, err := client.Dial(client.Options{
		HostPort:  "localhost:7233",
		Namespace: "test",
		ConnectionOptions: client.ConnectionOptions{
			TLS: &tls.Config{Certificates: []tls.Certificate{cert}},
		},
	})
	if err != nil {
		log.Fatalln("unable to create Temporal client", err)
	}
	defer c.Close()

Is there somethings that I have missed here? I have checked locally that the certificates loaded in the go-client are verified by the ca.cert in set in the cluster.