Best way to send email notification

Community Support
1

I was looking at example mentioned here https://github.com/temporalio/samples-java/blob/f107c79e29453ecd43dce522dfd1c980439b3762/core/src/main/java/io/temporal/samples/listworkflows/CustomerWorkflowImpl.java

   Workflow.retry(
        customerRetryOptions,
        Optional.of(expiration),
        () -> {
          customerActivities.getCustomerAccount(customer);
          customerActivities.updateCustomerAccount(customer, message);
          customerActivities.sendUpdateEmail(customer);
        });

As we can see here customerActivities.sendUpdateEmail(customer); , we are sending email to customer.

Let’s say if sending email fails, then will it run (retry) the whole workflow again or just activity sendUpdateEmail method (as retries are at activity level)

Alternatively, will be be best to create child workflow in abandon state here to send email so that this workflow can complete and sending email will be separate workflow?

2

Hello @shivamtiwari2009

Let’s say if sending email fails, then will it run (retry) the whole workflow again or just activity sendUpdateEmail method (as retries are at activity level)

Do you mean if the activity fails (ActivityTaskFailed) because e.g. the max number of attempts is reached or it throws a non-retryable failure? In that case, the Workflow.retry block will be retried according to the customerRetryOptions, and if the timeout is not reached.

Workflow executions do not retry by default.

Alternatively, will be be best to create child workflow in abandon state here to send email so that this workflow can complete and sending email will be separate workflow?

What are the requirements?

I don’t think you need a child workflow, the easiest way I think is just to execute your activities within the workflow method.

Something like:

  public void execute(Customer customer, String message) {

      customerActivities.getCustomerAccount(customer);
      customerActivities.updateCustomerAccount(customer, message);
      customerActivities.sendUpdateEmail(customer);

  }

You could have different activity options (and retry options) for each method if required.

Antonio

3

Thanks @antonio.perez for response. Here is my usecase

Say in above code I have many customers and updating each customer

private void updateCustomers()
        for (customer : customers) {
           execute(customer, "some message");
        }
}

public void execute(Customer customer, String message) {

      customerActivities.getCustomerAccount(customer);
      customerActivities.updateCustomerAccount(customer, message);
      customerActivities.sendUpdateEmail(customer);

  }

Now, since customerActivities.sendUpdateEmail(customer); is going to be blocking call, and say email server is down or it failed to send email to customer. Activity will retry (until max attempt or indefinite). But this will halt other customers to be updated.

My question is what will be best way to perform above operation where I can update customers and email should be sent to all customers regarding update.

4

I see,

One way is to start child workflow executions in parallel, one for each customer. Depending on the number of customers, you have to consider doing it in batches and continueAsNew every x number of customers to prevent the event history from hitting the limit.

Take a look at these examples

If you don’t need to wait in the parent for the child workflows to complete, you can start the child workflows in abandon mode, as you have mentioned.

Let me know if it helps,
Antonio

5

Thanks @antonio.perez

As you mentioned we have to do it in batches, that means if I create 10 child workflows for 10 customers then before I process next batch I have to wait for 10 child workflows to finish first. Now let’s say one of the these 10 child workflow failed to send email and say I have used infinite retry. Now this means I cannot process next batch because one of the workflow will not finish. What should be done in this case? We can’ t just create new workflows as you mentioned to avoid event history limit

6

@shivamtiwari2009

I am not sure if you have had the chance to see this example https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/slidingwindow

Do you have to wait, in the parent workflow, for the child workflows to finish? If not, maybe you can start the child workflows in abandon mode and call continue as new after finishing each batch.

Antonio

7

Thanks @antonio.perez After checkout out above link for slidingWindow I understand how it is working and how are we processing records in batches. My only question now is since here we are processing each records as workflow and there will be many workflows depending on number of records. How to handle rate limiting of sending emails? Is this something I have to handle at sending email service layer where I wait for certain time before I send email? It is possible that all parallel workflows are hitting rate limit and will get stuck for long time.

8

Use a separate task queue and associated worker for “send-email” activity. And configure a global rate limit for that task queue.

9

Thanks @maxim this will work

10

@maxim I was trying to implement separate task queue but then realized I need to have task queue per tenant. This way we are not rate limiting across tenant by having single task queue send-email. Basically we need something send-email-tenant-id1, send-email-tenant-id2 etc.

After reading docs it seems like we have to associate worker corresponding to each task queue. I am using spring boot does below code work

@Component
public class TemporalRegistration {
 
 public static String EMAIL_TASK_QUEUE = "send-email-";
 private static String RATE_LIMIT = 10;

  @EventListener
  public void onApplicationEvent(ContextRefreshedEvent event) {

   for (tenant: tenantsList) {
        final Worker emailWorker = workerFactory.newWorker(EMAIL_TASK_QUEUE + tenant.getId() ,
                    WorkerOptions.newBuilder().setMaxTaskQueueActivitiesPerSecond(RATE_LIMIT).build());
          emailWorker.registerWorkflowImplementationTypes({list of workflows });
          emailWorker.registerActivitiesImplementations({list of activities});
   }
  }
    workerFactory.start();
}

And then I can simply start workflow per tenant

final WorkflowOptions workflowOptions =
        WorkflowOptions.newBuilder()
            .setTaskQueue(EMAIL_TASK_QUEUE + tenantId)
            .setWorkflowId(workflowId.toString())
            .build();
    final EmailWorkflow workflow =
        workflowClient.newWorkflowStub(
            EmailWorkflow.class, workflowOptions);
    WorkflowClient.start(() -> workflow.sendEmail();

Do you think this is right approach ?

11

It depends on the number of tenants. It works for up to hundreds, not for thousands.

12

Is there any alternative approach if we have 1000’s of tenants?

13

Why do you rate limit individual tenants and not the overall email-sending rate?

14

This is more of generic thing I was thinking to solve similar problem where each tenant is making call to external service and each tenant external service has rate limit. So I thought I can use similar pattern there too and in sending email

15

Using Temporal for building a multi-tenant offering is a very common use case. We plan to address fairness, priority, and rate-limiting issues in the future. But it is a pretty long term goal.

16

So as of now there is no workaround for this problem?

17

One option would be using an external rate-limiting system and failing activities that exceed the limit. It would create additional churn but could work.

18

Yeah, I was thinking on similar grounds, I will do some poc with this and see how it goes.

Thanks for providing direction

19

You can also use a mixed approach. A task queue for each large tenant and a shared task queue for all the small ones.

20

yeah this also good idea. I think this will work in short term as you mentioned this will work for 100’s of tenants