Configuring namespace specific TLS certificates dynamically

Community Support
1

Hi Temporal team!

First of all, thank you for creating such a powerful system and keeping it open source.

I have been working on securing connections from Temporal clients. Following the customization-samples I adopted a solution where each client connecting to a namespace has its own cert issued against a namespace specific CA (for easy revocation without affecting other clients) and a custom authorizer validates that they are indeed accessing the designated namespace.

When configuring certs for each namespace I see that it is configured under global/tls

However, we have a need to create namespaces dynamically, and generate certs for them dynamically. We will be deploying Temporal on K8s and I found that dynamic configs can be updated without cycling the pods. It seems though that dynamic configs are limited to these configuration keys. Is this a correct assumption?

Would it be possible to allow modifying tls/frontend/hostOverrides dynamically (shown in code below)? The certs on the pods would be injected using Vault sidecars.

tls:
    internode:
        server:
            requireClientAuth: true
            certFile: /etc/temporal/config/certs/cluster/internode/cluster-internode.pem
            keyFile: /etc/temporal/config/certs/cluster/internode/cluster-internode.key
            requireClientAuth: true
            clientCaFiles:
                - /etc/temporal/config/certs/cluster/ca/server-intermediate-ca.pem
        client:
            serverName: internode.cluster-x.contoso.com
            rootCaFiles:
                - /etc/temporal/config/certs/cluster/ca/server-intermediate-ca.pem
    frontend:
        server:
            requireClientAuth: true
            certFile: /etc/temporal/config/certs/cluster/internode/cluster-internode.pem
            keyFile: /etc/temporal/config/certs/cluster/internode/cluster-internode.key
            clientCaFiles:
                - /etc/temporal/config/certs/cluster/ca/server-intermediate-ca.pem
        client:
            serverName: internode.cluster-x.contoso.com
            rootCaFiles:
                - /etc/temporal/config/certs/cluster/ca/server-intermediate-ca.pem
        hostOverrides:
            accounting.cluster-x.contoso.com:
                certFile: /etc/temporal/config/certs/cluster/accounting/cluster-accounting-chain.pem
                keyFile: /etc/temporal/config/certs/cluster/accounting/cluster-accounting.key
                requireClientAuth: true
                clientCaFiles:
                    - /etc/temporal/config/certs/client/ca/client-intermediate-ca-accounting.pem
            development.cluster-x.contoso.com:
                certFile: /etc/temporal/config/certs/cluster/development/cluster-development-chain.pem
                keyFile: /etc/temporal/config/certs/cluster/development/cluster-development.key
                requireClientAuth: true
                clientCaFiles:
                    - /etc/temporal/config/certs/client/ca/client-intermediate-ca-development.pem

Thank you for your time and suggestions.

2

I should clarify that we are looking for a way to dynamically configure only the hostOverrides section, not the entire tls or frontend section which would be more difficult to dynamically configure as they are used in internode exchange.

3

You can use WithTLSConfigFactory server option to inject your own plugin. That plugin then can dynamically load updated configuration and corresponding certificates. This way you can add and remove entries in the hostOverrides section without rebooting the nodes.

1 Like
4

Thanks for the pointer, much appreciated. From a brief investigation it seems that we could extend localStoreCertProvider and add the desired certificate loading policy. We already have a custom Temporal server to use the authorizer, so adding the TLS config factory option is viable.

5

Great! Let us know if you run into any problem.

1 Like