We’re using Temporal for email/sms campaign sending, and we need to do this as quickly and efficiently as possible. Each campaign could be up to 5 million recipients. Here’s our current setup.
- Start a parent workflow that gets batches of ~5000 recipients from the database, and then starts child workflows for each batch of 5k. Like this:
let lastId: string | undefined;
let i = 0;
const childWorkflowPromises = [];
while (true) {
const campaignRecipientIdsBatch = await getRecipientBatchActivity(
{ campaignId, pageSize: maxRecipientsPerWorkflow, lastId }
);
if (campaignRecipientIdsBatch.length === 0) {
break;
}
lastId = campaignRecipientIdsBatch[campaignRecipientIdsBatch.length - 1];
childWorkflowPromises.push(
executeChild(sendRecipientBatch, {
args: [{ teamId, recipientIds: campaignRecipientIdsBatch }],
workflowId: `send-campaign-${campaignId}-batch-${i}`,
}),
);
i++;
}
await Promise.allSettled(childWorkflowPromises);
- The child workflow will take that batch, and start activities in parallel for each one of those recipient to do the actual sending.
export async function sendRecipientBatch({
teamId,
recipientIds,
}: {
teamId: string;
recipientIds: string[];
}) {
await Promise.allSettled(
recipientIds.map((recipientId) =>
processRecipientActivity({ teamId, recipientId }),
),
);
}
- The processRecipientActivity carefully sends an email/sms to that recipient. The way it does that is that if we fail to send for any reason, we wait 10 minutes for a retry, and first thing we do on the retry is check if we’ve gotten a successful send event already for this recipient, and if so we don’t continue.
- Since we’re rate limited on how many sends we can per second by our downstream providers, we are setting maxActivitiesPerSecond, so that even though we will have hundreds of thousands of pending activities at a time, we only try to process them at maxActivitiesPerSecond.
Does this sound like a good pattern? Any suggestions of how we should do things better or issues that might come up? We need to send as fast as possible, but we also need to be sure that we don’t double send.
@maxim would love your feedback, I’ve enjoyed reading your input on other people’s problems here. Thanks!