Best way to model video export workflow with retries and long-running activities

Community Support
1

Hi everyone,

I’m evaluating Temporal for a backend workflow in a video editor-style application where users upload clips, apply simple edits, and then request an export. The product is not meant to be a full free capcut clone, but the workflow is similar in the sense that a user may start a video export, leave the app, and come back later to see whether the export completed, failed, or needs to be retried.

My current idea is to model the export process as a Workflow with multiple Activities:

  1. Validate the uploaded media files

  2. Generate preview metadata/thumbnails

  3. Send the render job to a processing service

  4. Poll or wait for the render result

  5. Upload the final exported video to storage

  6. Update the project/export status in the database

The part I’m unsure about is how to handle long-running render jobs and retries correctly. Some exports may take a few minutes, while larger files could take much longer. If the external rendering service fails or times out, I want Temporal to retry safely without accidentally creating duplicate export jobs or overwriting the wrong project status.

For this kind of video editor/export workflow, would you recommend keeping the render job as one long Activity with heartbeat/timeouts, or splitting it into smaller Activities with signals/callbacks from the rendering service?

Also, what is the recommended pattern for making the external render request idempotent so that Temporal retries don’t create duplicate video exports?

Any guidance on Workflow design, Activity retries, heartbeat timeout settings, or idempotency patterns would be really helpful.

4

Hi Alisha,

Temporal is well-suited for your use case — but Temporal alone won’t solve all your problems. You still need a good design; otherwise, you’ll end up fighting Temporal instead of letting it work in your favor.

For this kind of video editor/export workflow, would you recommend keeping the render job as one long Activity with heartbeat/timeouts, or splitting it into smaller Activities with signals/callbacks from the rendering service?

As a rule of thumb: every side effect should be its own activity.

Here’s why. Say you have side effect #1 and side effect #2, and you run them inside a single activity. If #2 fails, Temporal retries the whole activity — which means #1 runs again. There’s no way around that; activities are the unit of retry.

Split them into two activities, and the picture changes: if #2 fails and retries, #1 is not re-executed. Temporal guarantees that.

One important caveat though: “one side effect per activity” is necessary but not sufficient. Activities can still retry after the side effect has already succeeded — for example, if the worker crashes between completing the side effect and reporting completion back to the server, Temporal has no way to know the work was done, so it retries.

This means the side effect itself must be idempotent: use an idempotency key, a dedup check, a conditional write, or a “has this already happened?” lookup before doing the work. Without that, you’ll occasionally double-charge a card, double-upload an asset, or double-send a notification — and these bugs only show up under failure, which makes them painful to track down.

So for your render pipeline: identify the non-idempotent or expensive steps (uploading the final asset, charging the user, notifying downstream systems, kicking off the render itself), give each its own activity, and make each one idempotent. Use heartbeats on the long-running render activity so Temporal can detect a stuck worker, and use signals if the rendering service needs to push progress or completion back into the workflow.

About the idempotent key: in your case, it can be anything like ${userId}-${projectId}-${fileId}. You decide your idempotent key format.