Use temporal to replace transactional outbox pattern

Tech Comparisons
1

I’m new to temporal and there is a use case I have in mind that I want to check with the community if it is a fit.
I currently have application logic to write events into postgres and a separate process to read from a logical replication slot to push these events to an event bus. So in short, it’s a CDC process/transactional event outbox pattern. This pattern gives us latency around sub 500ms(p95) for each event, more than 10k events per second sent to the event bus and at least once event delivery to the bus (avoiding the common pitfall of dual writes to db and bus).
I’m wondering if I can use a workflow to simplify such set up to achieve the same or better non functional characteristics?

2

Hi Shine, welcome to the forum! I assume writing the event to Postgres is in a transaction with another DB update? Using workflows is a much simpler alternative to transactional outboxes—you just do:

async function myWorkflow(info) {
  await dbUpdateLocalActivity(info)
  await publishEventLocalActivity(info)
}

  • Latency: Assuming Postgres and the event bus are nearby, it should normally complete within 500ms.

  • At least once delivery: Yes

  • Throughput: 10k events/s is definitely possible on Temporal Cloud (we’ve tested up to a million state transitions per second), and I think you could reach it with a self-hosted Temporal Cluster if it’s tuned well, along with a large enough Cassandra cluster.

Some more efficient approaches would be to start a workflow for each batch of events or using a long-running activity with heartbeats to record progress and recover from a crash. There are also ways to parallelize the processing of the replication log by partitioning the events based on their ID.

Eager workflow start also helps reduce the load and latency but that’s currently only available with latest server and the Java SDK.