Hey folks, I was wondering if someone could answer a couple of questions I’ve got regarding mTLS setup + .NET SDK.
I’ve got a CA cert for temporal and have generated and signed cluster and client certs. The communication between temporal services seem fine, however when I setup the certs for my client connection, it fails:
services.AddTemporalClient(options =>
{
options.TargetHost = temporalConfig.Host;
options.Namespace = "my-namespace";
options.Tls = new TlsOptions
{
ClientCert = File.ReadAllBytes("temporal-client-cert.pem"),
ClientPrivateKey = File.ReadAllBytes("temporal-client.key"), // Also tried as a .pem
};
});
And the error:
System.InvalidOperationException: Connection failed: `get_system_info` call error after connection: Status { code: Unknown, message: "transport error", source: Some(tonic::transport::Error(Transport, hyper::Error(Io, Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }))) }
at Temporalio.Bridge.Client.ConnectAsync(Runtime runtime, TemporalConnectionOptions options)
at Temporalio.Client.TemporalConnection.GetBridgeClientAsync()
at Temporalio.Client.TemporalConnection.ConnectAsync(TemporalConnectionOptions options)
at Temporalio.Client.TemporalClient.ConnectAsync(TemporalClientConnectOptions options)
at Temporalio.Extensions.Hosting.TemporalWorkerService.ExecuteAsync(CancellationToken stoppingToken)
at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
I created my certs by looking at the generate-test-certs script for insight and used the client certs.
Have I missed something out/ done something wrong?
This is a self-hosted cluster and not cloud? Usually if you have a self-hosted cluster, the server-side certificate (unrelated to mTLS, just the normal server TLS part) is self-signed which means you usually need to set the CA manually. Or are you installing the server CA on the system? How are you configuring the server’s TLS?
Can you do things via the CLI and this is just .NET specific, or does this issue occur on both?
Yes this is a self hosted cluster, though, my apologies, I should have mentioned that I’m currently running the temporal services via docker compose with mTLS enabled on my machine. I use dockerfiles and install the server CA certificates on the system and I am able to use the cli, specifying the location of the ca certificate and cluster certificate & key to create a namespace, but using the client (or even cluster) certificates from the .NET SDK don’t seem to work.
I did notice that in the .NET SDK the docs, albeit quite short, mention connecting using mTLS but only to temporal cloud. Is connecting to self hosted any diffferent?
There may be a current issue with using CAs loaded from the system in 1.3.0 and 1.3.1 due to a Rust dependency we have that had a bug in it (Tonic 0.12.1). We have a simple issue open to fix it for next release, but in the meantime, can you confirm that .NET SDK 1.2.0 works for you? Alternatively you can manually set the ServerRootCACert value to the CA cert you are loading on the system.
We will look to prioritize a 1.3.2 release with this dependency.
I tried using v1.2.0 and added the ServerRootCACert property that is used on the server but I still seem to have the same issue:
services.AddTemporalClient(options =>
{
options.TargetHost = temporalConfig.Host;
options.Namespace = "my-namespace";
options.Tls = new TlsOptions
{
ClientCert = File.ReadAllBytes("temporal-cluster-cert.pem"),
ClientPrivateKey = File.ReadAllBytes("temporal-cluster.key"),
ServerRootCACert = File.ReadAllBytes("server-root-ca.cert")
};
});
I am not aware of any issues beyond those. We have successfully connected to our own cloud and to self-hosted clusters without issue with the .NET SDK.
Does this only occur for the worker, or all uses of the client even direct uses? How are you initializing the worker? There are two overloads of AddHostedTemporalWorker, one that uses the DI’d client from AddTemporalClient like you have here, another that implicitly creates its own (see the README here for explanation).
It fails for direct uses of the client as well, though I realised I had mistakenly overridden the temporal client that was being created by AddTemporalClient but now I’m getting the following error:
Connection failed: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, Custom { kind: InvalidData, error: InvalidCertificate(NotValidForName) }))
This happens with version 1.2.0 and 1.3.1
I’m wondering if I have maybe made a mistake when generating my certs. This is the process I followed:
# Generate a private key and a certificate for certificate authority
openssl genrsa -out ca.key 4096
openssl req -new -x509 -key ca.key -sha256 -subj "/C=GB/ST=Greater London/O=My Company Inc." -days 3650 -out ca.cert
# Generate a private key and a certificate for cluster
openssl genrsa -out cluster.key 4096
openssl req -new -key cluster.key -out cluster.csr -config cluster.conf
openssl x509 -req -in cluster.csr -CA ca.cert -CAkey ca.key -CAcreateserial -out cluster.pem -days 3650 -sha256 -extfile cluster.conf -extensions req_ext
# Generate a private key and a certificate for clients
openssl req -newkey rsa:4096 -nodes -keyout client.key -out client.csr -config client.conf
openssl x509 -req -in client.csr -CA ca.cert -CAkey ca.key -CAcreateserial -out client.pem -days 3650 -sha256 -extfile client.conf -extensions req_ext
My client.conf is as follows:
[req]
default_bits = 4096
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[dn]
C = GB
ST = Greater London
O = My Company Inc.
CN = localhost
[req_ext]
subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost
IP.1 = ::1
IP.2 = 127.0.0.1
I’m using the client.pem, client.key and ca.cert in my TlsOptions
Connection failed: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, Custom { kind: InvalidData, error: InvalidCertificate(NotValidForName) }))
@Chad_Retz do you know if the above error refers to the Common Name in the cert? Or is this an issue with the Tonic dependency?
This usually means that yes the CN or SAN on the cert does not match the host you’re connecting to. What is host:port you are connecting to? If you set that as the Domain TLS option, does the connection work?
I’m connecting to localhost:7233 which is set on the SAN and CN of the cert signed by the CA.
I on a whim I removed the section in my Dockerfile which installs the server ca, and that actually did the trick. I must have been doing it incorrectly, but now just setting up the CAs used by our proxy and then supplying the ServerRootCA in the options allows it to connect.
On an unrelated note though, with using the .NET SDK, how are we supposed to set an initial ApiKey? I’m able to connect using mTLS and use SSO now, but my service which has my workflow code would require it’s own ApiKey, and from the looks of things we would need to build in our own implementations to rotate tokens?
@Chad_Retz are you able to advise on the above?
Can start a new thread on unrelated things if needed
Initial API key is set via the ApiKey connection option when you first create/connect the client. Not to be confused with the client’s ApiKey property on the client. Due to how the worker works written in Rust, you have to use the ApiKey property setter on the client to update it (i.e. updates to this value have to be pushed, they are not pulled).
1 Like