AWS Lambda (dotnet 8) - connection/certificate issue

Community Support
,
1

Greetings - we’ve just created an AWS Lambda which runs some dotnet 8 code that signals workflows with the Temporal Dotnet SDK. However I think we’ve hit a rust/platform issue. The Temporal client fails to connect with the error below. Credentials are fine, the code works on alternative dotnet environments, and Lambdas deployed the same way but written with the Python SDK do not seem to be affected. I think the default AWS dotnet lambda runtime image is here - has anyone encountered this? What are our options to circumvent the issue?

System.InvalidOperationException: Connection failed: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, Custom { kind: InvalidData, error: InvalidCertificate(UnknownIssuer) }))
   at Temporalio.Bridge.Client.ConnectAsync(Runtime runtime, TemporalConnectionOptions options)
   at Temporalio.Client.TemporalConnection.GetBridgeClientAsync()
   at Temporalio.Client.TemporalConnection.InvokeRpcAsync[T](RpcService service, String rpc, IMessage req, MessageParser`1 resp, RpcOptions options)
   at Temporalio.Client.WorkflowService.StartWorkflowExecutionAsync(StartWorkflowExecutionRequest req, RpcOptions options)
   at Temporalio.Client.TemporalClient.Impl.StartWorkflowInternalAsync[TWorkflow,TResult](StartWorkflowInput input)
   at Temporalio.Client.TemporalClient.Impl.StartWorkflowAsync[TWorkflow,TResult](StartWorkflowInput input)
   at Temporalio.Client.TemporalClient.StartWorkflowAsync(String workflow, IReadOnlyCollection`1 args, WorkflowOptions options)
  <...snip...>
2

The .NET and Python SDKs use the same TLS logic. Is this using self-hosted or cloud? At first glance, the failure looks like it can’t validate the CA of the server cert. This is most likely caused by the image/system having out of date CA certs. I don’t see in the lambda bootstrap script where it is updating the CA certs on the system.

3

This is cloud, and using self-signed certs created with tctl. You are right it sure does looks like an issue with the certs, however the exact same ones do work when running off my local machine, or when running using python on an AWS Lambda…

4

I am saying it may not be an issue with the client certs with mTLS but rather the server certificate we send and your system’s inability to validate it against its trusted set of CAs. Your local machine has updated CA certs but the AWS machine may not. I do not know for sure if this is the case, but it is commonly the case when the exact same client code/options work on one machine but not another and you get these unknown issuer failures. It is clearly environment specific and not code related. You may need to find a way to update the system certificates on the AWS Lambda instance.

5

Yeah - at the moment we’re rolling a new Lambda image that does work. Maybe in the future when we know a bit more about this stuff, we can find a more light-weight way to do it.