PoC: Dynamic Activities and one worker per activity

Community Support
,
1

Hello everyone,

I am working on a PoC to validate an idea: having a workflow that uses activity names generated dynamically by each application sending the workflow to Temporal. For example, application1 sends a workflow with an activity named “activity1”, while application2 calls the same workflow, this time with an activity named “activity2”.

So far, the best I have come up with was by using Dynamic Activities.

The final setup I considered was:

  • 1 worker that manages the workflow (it’s a typed workflow that uses Workflow.newUntypedActivityStub for the calls to the individual dynamic activities)

  • 1 worker per activity type/task queue:

        ActivityOptions activityOptions =
              ActivityOptions.newBuilder()
                      .setTaskQueue(activityName)
                      .setScheduleToCloseTimeout(Duration.ofSeconds(10))
                      .build();

      ActivityStub activity =
              Workflow.newUntypedActivityStub(
                      activityOptions);
      Object v = activity.execute(activityName, Object.class, context);

The DynamicActivity implementation class loads the class to execute based on the activity name passed in the workflow.

There is more logic than I explained in the post (we are actually using an own DSL for the workflow), but the description above summarizes it.

The infrastructure is based on K8s, therefore, among the other things, I know which activities consume the most and can set an autoscaling on cpu/memory etc.

Now, I would like to ask if there are any known limitations on:

  • the approach proposed: is it okay? will I get into issues in the long run?
  • are there any limitations on number of task queues?
  • namespaces: are there any limitations on how many we can create? I would like to eventually have a multi-tenant environment, where each tenant potentially matches 1 namespace. Is this an option? Any other idea is welcome.

Thanks a lot in advance.

2

@maxim I know that this is very specific, but I wonder if you could help me on some of the points described above? :slight_smile: Thanks a lot!

3

Sorry for the late reply, missed this question.

  • the approach proposed: is it okay? will I get into issues in the long run?

Yes, it looks reasonable.

  • are there any limitations on number of task queues?

I don’t think you need a task queue per activity type unless you want to run each activity in a separate process.

  • namespaces: are there any limitations on how many we can create? I would like to eventually have a multi-tenant environment, where each tenant potentially matches 1 namespace. Is this an option? Any other idea is welcome.

We don’t know the limit. We ran with 200 namespaces in production. I recommend testing this. Let us know if you run into any issues with the large number of namespaces.

4

Hi @mbu can i know how you have loaded the class based on the activity name?