PermissionDenied using .Net SDK and CLI works just fine

I have deployed Temporal on Kubernetes using the Helm chart and the default settings.
When I use the CLI it works:

> temporal env set alp.address <HOST>:443

> temporal operator cluster health --env alp --tls                                     
SERVING

> temporal operator namespace --env alp --tls list
  NamespaceInfo.Name                    temporal-system
  NamespaceInfo.Id                      32049b68-7872-4094-8e63-d0dd59896a83
  NamespaceInfo.Description             Temporal internal system namespace
  NamespaceInfo.OwnerEmail              temporal-core@temporal.io
  ...

I was also able to create a namespace with the CLI.

Now, in my .Net code, I use

builder.Services.AddTemporalClient(opts =>
{
    opts.TargetHost = "<HOST>:443";
    opts.Namespace = "<NAMESPACE>";
    opts.Tls = new TlsOptions();
});

Any operation I try leads to:

System.InvalidOperationException: Connection failed: Server connection error: tonic::transport::Error(Transport, Os { code: 13, kind: PermissionDenied, message: "Permission denied" })
          at Temporalio.Bridge.Client.ConnectAsync(Runtime runtime, TemporalConnectionOptions options)
          at Temporalio.Client.TemporalConnection.GetBridgeClientAsync()
          at Temporalio.Extensions.Hosting.TemporalWorkerService.ExecuteAsync(CancellationToken stoppingToken)
          at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)

Evidently what the CLI and the .Net SDK do is different.
I looked at the logs, but can’t find anything.

My setup is using Istio to terminate the TLS connection and forward to Temporal frontend. I have used that setup with my own gRPC services, there is no problem there, it is basic HTTP(2) proxying, and again the CLI works just fine so it must be on the .Net SDK side.

This is on Ubuntu 22.04 if that matters.

At first glance it looks like the OS is giving permission denial. Assuming these occur on the same machine, I am unsure why the Go connection succeeds (lang CLI is in) but the Rust connection fails (lang the underlying .NET code for this is in).

Can you confirm it works without your proxy? How are you configuring the CA cert the TLS connection validates the server cert against? If it is installed as a system cert, I wonder if the Rust side is not loading the system CA correctly. Can you confirm your .NET SDK version and if >= 1.3.0 can you try 1.2.0 and see if the behavior persists? You can also try putting the system CA cert into the TLS options, though we do expect system CAs to work (we had an issue recently with one of our library upgrades here, e.g. this, so if this works in 1.2.0 we’ll know this is the issue).

1 Like

Thank you. I was so focused on the gRPC aspect that I missed that the error was client side.

I downgraded to 1.2.0 .Net SDK and it works perfectly, so there was a regression in 1.3.x.

1 Like

Yes, this was an issue with Tonic which they have now fixed. I have opened an issue to upgrade Tonic. If you need this immediately, you can build the SDK after having run cargo update in src/Temporalio/Bridge.