Getting error TransportError (InvalidCertificateEncoding) when using SSL

Community Support
1

I’m trying to connect my typescript worker to temporal server using TLS and I’m getting the following error back:

[DEBUG] connecting to <IP ADDRESS>:<PORT>
[DEBUG] connected to <IP ADDRESS>:<PORT>
[ERROR] TransportError: tonic::transport::Error(Transport, hyper::Error(Connect, Custom { kind: InvalidData, error: InvalidCertificateEncoding }))

And this is the log from temporal frontend:

attempted incoming TLS connection","address":"XXX.XX.XXX.X:42592
returning TLS config for connection","address":"XXX.XX.XXX.X:42592

It seems to be something related to how ts-sdk uses core-sdk (rust), but I’m not sure.

*** In addition, I have another Ruby worker (coinbase temporal sdk) connected to the same Temporal instance using SSL with no issues. ***

TCTL works fine on my container using the same cert:

tctl --tls_server_name=<My Server Name> --address=<IP>:<PORT> --tls_ca_path=./ssl/certs/ca.crt --tls_cert_path=./ssl/certs/tls.crt --tls_key_path=./ssl/certs/tls.key cl h
temporal.api.workflowservice.v1.WorkflowService: SERVING

This is my worker’s code:

import {
  DefaultLogger,
  NativeConnection,
  Runtime,
  Worker,
} from '@temporalio/worker';
import type {LogLevel} from '@temporalio/worker';
import {test1Activities, test2Activities} from './all-activities';
import * as dotenv from 'dotenv';
import fs from 'fs-extra';

dotenv.config();
const logLevel = (process.env.LOG_LEVEL || 'DEBUG') as LogLevel;
const logger = new DefaultLogger(logLevel);

async function run() {
  const temporalPort = process.env.TEMPORAL_PORT || 7233;
  const temporalHost = process.env.TEMPORAL_HOST || 'localhost';

  logger.info(`Connecting to ${temporalHost}:${temporalPort}`);
  Runtime.install({
    logger,
    telemetryOptions: {
      metrics: {
        prometheus: {bindAddress: '0.0.0.0:9090'},
      },
      logging: {forward: {level: 'DEBUG'}},
    },
  });

  const caFile = process.env.TEMPORAL_CA_CERT_PATH || '/app/ssl/certs/ca.crt';
  const crtFile =
    process.env.TEMPORAL_CLIENT_CERT_PATH || '/app/ssl/certs/tls.crt';
  const keyFile =
    process.env.TEMPORAL_CLIENT_KEY_PATH || '/app/ssl/certs/tls.key';

  const connection = await NativeConnection.connect({
    address: `${temporalHost}:${temporalPort}`,
    tls: {
      serverRootCACertificate: fs.readFileSync(caFile),
      serverNameOverride: '<MY Server Name Here>',
      clientCertPair: {
        crt: fs.readFileSync(crtFile),
        key: fs.readFileSync(keyFile),
      },
    },
  });

  const test1Worker = await Worker.create({
    connection,
    workflowsPath: require.resolve('test1/workflows'),
    activities: test1Activities,
    namespace: process.env.TEMPORAL_NAMESPACE,
    taskQueue: 'test1',
    bundlerOptions: {ignoreModules: ['@temporalio/client']},
  });

  const test2Worker = await Worker.create({
    connection,
    workflowsPath: require.resolve('test2/workflows'),
    activities: test2Activities,
    namespace: process.env.TEMPORALITE_NAMESPACE,
    taskQueue: 'test2',
    bundlerOptions: {ignoreModules: ['@temporalio/client']},
  });

  await Promise.all([test1Worker.run(), test2Worker.run()]);
}

run().catch((err) => {
  logger.error(err);
  process.exit(1);
});

Some more info:

  • TS worker runs on Ubuntu 20.04.6 LTS
  • Temporal server 1.20.2
  • Temporal Typescript SDK: 1.7.2

Appreciate any help. :pray:

2

Can you please verify that your cert/key files have the following formats (simply open them with a text editor)?

caFile:

-----BEGIN CERTIFICATE-----
... a few lines of alphanumeric data ...
-----END CERTIFICATE-----

crtFile:

-----BEGIN CERTIFICATE-----
... a few lines of alphanumeric data ...
-----END CERTIFICATE-----

keyFile:

-----BEGIN PRIVATE KEY-----
... a few lines of alphanumeric data ...
-----END PRIVATE KEY-----
3

Also, can you please try creating a client connection with the same certificate/key? For example:

  const connection = await Connection.connect({
    address,
    tls: {
      serverNameOverride: ...,
      serverRootCACertificate: ...,
      clientCertPair: {
        crt: fs.readFileSync(...),
        key: fs.readFileSync(...),
      },
    },
  });
4

caFile and crtFile look the same, the only difference is that keyFile is like:

-----BEGIN RSA PRIVATE KEY-----
... a few lines of alphanumeric data ...
-----END RSA PRIVATE KEY-----

instead of:

-----BEGIN PRIVATE KEY-----
... a few lines of alphanumeric data ...
-----END PRIVATE KEY-----
5

Also, I just tested a Client connection (not NativeConnection) using the same certs and I was able to start a workflow with no issue.

So, I believe I can’t Connection instead of NativeConnection in my worker, correct? :thinking:

6

No, you can’t. NativeConnection are backed by Core SDK (ie. Rust code, required for Workers), while Connection uses a pure JavaScript implementation (usable only with Client).

Ok, that’s pretty much what I was expecting… BEGIN RSA PRIVATE KEY indicates a PKCS#1 formated key. Can you try converting it to PKCS#8 unencrypted PEM format?

There are various tools you can use for this, but if you have access to OpenSSL, this would be the syntax to use:

# openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in tls.key -out tls.pem
7

I tried this:

openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in ssl/certs/tls.key -out newtls.key

And then passed newtls.key to tls.clientCertPair.key conf, but no luck.
I got the same InvalidCertificateEncoding error.