Design Review: Job-Based Execution and External Business State with Temporal

Community Support
1

Hey

We’re integrating Temporal across our business, with multiple repositories each owning one or more workflows.

As the number of workflows grows, we’re running into two recurring challenges:

  • We want to avoid persisting client/business-sensitive data in Temporal history and storage.
  • We need a consistent recovery mechanism for cases where Activity retries are exhausted and manual intervention/retry is required.

Since workflows are distributed across multiple repositories, each service currently has to solve these concerns independently. This is leading to duplicated code and slightly different patterns across workflows.

To address this, we’re considering introducing a common Job-based execution model around Temporal.

Before a workflow can be started, a Job is created containing the workflow type, input, current status, and requested action. A central scheduler processes these Jobs and translates the requested action into the appropriate Temporal operation.

Conceptually:

Service → Create Job → Scheduler → Temporal → Workflow

This gives us a single application-level entry point for starting and managing different types of workflows, instead of every calling service interacting with Temporal directly.

Along with the Job, we’re considering a JobState table to persist the business state produced as the workflow progresses.

A JobState entry represents a successful workflow/activity checkpoint. For example:

Job
 └── JobState: INITIATED
        ↓
     Activity A ✓
        ↓
     JobState: A_COMPLETED
        ↓
     Activity B ✓
        ↓
     JobState: B_COMPLETED

Each Activity receives only the current jobStateId. The actual Job input and current state are loaded from our application database inside the Activity. After the Activity completes successfully, its updated state is persisted as a new JobState entry, and only the new jobStateId is returned to the workflow.

If an Activity fails, no new JobState checkpoint is created, so the latest JobState always represents the last successfully completed business step.

With this model, Temporal primarily receives identifiers such as jobId and jobStateId, while the actual client/business input and evolving workflow state remain in our application storage.

image
image1200×603 32.5 KB

To avoid implementing these patterns independently in every repository, we’re also considering a small shared framework on top of Temporal.

image
image822×463 26.2 KB

Recovery

For Activity failures, we would continue using Temporal’s normal Activity retry policy first.

If those retries are exhausted, instead of immediately failing the Workflow, we’re considering having the framework keep the Workflow alive for a configurable recovery period.



Activity fails
    ↓
Temporal Activity retries
    ↓
Retries exhausted
    ↓
Workflow waits for recovery
    ↓
 ┌──────────────┐
 │              │
RETRY         Timeout
 │              │
 ↓              ↓
Retry same     Workflow
Activity       FAILED

A user-initiated RETRY would be sent to the existing Workflow through a common Workflow Update. Since the wait happens around the failed Activity invocation, the framework can retry only that Activity and then allow the Workflow to continue normally.

If the recovery window expires, the Workflow is allowed to fail. A later retry request could start a new Workflow Execution, although we are still evaluating what the best Temporal-native approach is for recovery after a terminal Workflow failure.

Feedback we’re looking for

Before we go further with this approach, we’d particularly like feedback on whether we’re using Temporal in the intended way or introducing abstractions that work against its model.

  1. Job / Temporal boundary
    Does using Job as the application-level entry point/control layer while keeping Temporal responsible for durable execution and orchestration seem like a reasonable separation?
  2. External business state
    Is keeping business/client data in Job / JobState and passing primarily jobId / jobStateId through Temporal a reasonable approach? Are there important Temporal capabilities or guarantees we would lose by doing this?
  3. JobState checkpoints
    Does maintaining immutable successful Activity checkpoints externally make sense for this use case, or does this start duplicating state/checkpointing that Temporal should own?
  4. Recovery after Activity retries are exhausted
    Is keeping the Workflow alive with a durable wait and using a Workflow Update to retry the failed Activity an idiomatic Temporal pattern?
  5. Recovery after terminal Workflow failure
    If the recovery window expires and the Workflow becomes terminally failed, what would be the recommended Temporal approach for a later business-level retry? In particular, how would you approach continuing from externally persisted successful business state without effectively building a second orchestration engine?

The main thing we’d like to understand is where this design aligns with Temporal’s intended model and where we may be unnecessarily rebuilding functionality Temporal already provides.

Any architectural feedback or alternative patterns would be appreciated.