[Design Pattern] - Distributed Activity Execution in multiple Microservices

Community Support
1

Context:
We have around 15 microservices. Each one has it own responsability (for example, one is a directory of users, other keeps track of accounting things related with one user, other manages invoices, etc.).
On the other hand we have an “integration service” that “talks” with external services.
When a request comes from the external system, many different services can be involved because

  1. We have to check the user (for example, its license, the token, etc.etc.) and all that logic is in the directory service (and it is complex logic)
  2. We have to create some invoices (that depends on the user, and the “activity” for creating invoices is complex and is allocated in the “invoicing” service)
  3. We have to create some accounting transactions (that depends on the invoices, and which logic is allocated into the “accounting” service)

Design
In my mind, I though that a good approach could be:

  1. Start a worker in every service and register the activities that are responsibility of that service.
  2. Create a Workflow in the integration service (which doesn’t have any activity implementation) that orchestrates the activities involved
  3. When an external event comes to the integration service, a new workflow is fired and the “activity dance” starts orchestrated by this workflow.

Problem
I implemented this pattern… but with no success.

  1. If I register all activities in the same queue ➝ I get “Activity Type XXXXX is not registered with a worker. Known types are: YYY, ZZZZ”. The problem is that the worker of a microservice doesn’t have all the activity implementations because each one is located in its own microservice. At the end, it works because the retries… so finally Temporal can located the next worker and so on… but is an execution full of errors and very slow.
  2. If I register activities in different task queues ➝ It doesn’t work at all. It stops after the first Activity that has to be executed in other system (because it has a different task queue)

I have tried to find a way to accomplish that (because it is a must that every activity logic has to be located in the microservice that has the responsibility) but I haven’t found a way to do it (may be break the workflow in smaller workflows that are executed in every service with different task queues… but it dones’t look as an “elegant” way to do it).

The old system used http requests and it worked… the problem is that implement retries, etc. is very expensive and I though Temporal could do the trick.

Any suggestion? I thought that was the main use case for Temporal… but I think I’m wrong.

2

By default, an activity is scheduled in the task queue that has the same name as the workflow task queue. Note that workflow task and activity task queues are distnicts. Only their names are the same.

If you want to schedule activities on a different task queue you have to specify the task queue name explicitly in the ActivityOptions. Here is an example using Java SDK:

    ActivityOptions service1Options =
        ActivityOptions.newBuilder()
            .setTaskQueue("service1")
            .setScheduleToStartTimeout(Duration.ofSeconds(10))
            .build();
    Service1 service1 =
        Workflow.newActivityStub(Service1.class, service1Options);

To summarise: Each service should start a worker on its own task queue. And a workflow should specify the appropriate task queue when invoking activities and child workflows on that service.

1 Like
3

Thank you Maxim! It worked like a charm!!