Refactoring RabbitMQ-based microservices to Temporal: bridge pattern, task queues & worker layout questions

Community Support
,
1

Hi all,

I’m new to Temporal and am refactoring a set of Go microservices that currently communicate via RabbitMQ. I’d like Temporal to own the orchestration while still talking to those external services over RMQ.

Current flow

  1. MinIO event triggers the process.
  2. We ingest and parse a file from MinIO.
  3. We send a message to an external service over RabbitMQ and wait for its result (also via RabbitMQ).
  4. For each event in the file, we run a sub-process.
  5. Each event’s sub-process may again call an external service via RabbitMQ.

Proposed Temporal design

  • RabbitMQ ↔ Temporal bridge: a thin service that listens on RMQ queues and calls SignalWorkflow / SignalWithStartWorkflow.
  • Worker A: runs the parent workflow + child workflows (orchestrators).
  • Worker B: runs ingestion activities.
  • Worker C: runs activities for child workflows.
  • Parent workflow sends work to multiple task queues that these workers listen to.

Questions

  1. Is the thin “RMQ bridge” pattern (consumer → Signal/SignalWithStart) the recommended approach, or would you embed RMQ consumers inside the worker?
  2. Would you keep separate binaries/pods for each worker role, or one binary with multiple task queues? Pros/cons you’ve seen?
  3. For “per-event” processing, should I spin up a child workflow per event, or keep it as activities with a loop and maybe ContinueAsNew to manage history size?
  4. Any gotchas around idempotency/DLQ when using SignalWithStartWorkflow from RMQ messages?

Stack details: Go SDK, Temporal on Kubernetes, external services stay as-is (RMQ).

Thanks for any pointers or examples!

2

1, 2 : Temporal is not opinionated about how you organize your processes. Usual caveats of sharing CPU/Memory apply when you run multiple services and workers in a single process.
3. You mentioned that “sub-process” needs to communicate to external service through RabbitMQ. The response processing is easier to implement using a workflow than an activity.
4. Make sure to assign the same workflowID for message retries. Usually some sort of business id is recommended for workflowID.