Is there a centralized way in Temporal to notify all workers running on different JVM to suspend polling?

Community Support
,
1

Previously I found out from @maxim (over slack) that it is not possible to suspend polling on workers from a remote process. We are planning to use Temporal with multiple pods running in kubenetes with auto scaling, and need a way to suspend all workers on demand, is having a delegated workflow running on each pod and listen to a pubsub message to act on the “local” workers a feasible option? Any better approach you can suggest?

[Maxim Fateev]
It is not possible to suspend polling from a separate process.
We plan to add admin features for this, but there is no ETA.

2

I believe here is the related open issue for this request, and here is the related forum post.
The feature mentioned in the issue is still to be implemented. See if Maxims answer in the mentioned post would help for now.

3

Thanks @tihomir , some follow up questions,

Is it because a workflow can get picked up by a worker running in a different process/pod?

  1. how would this activity find the workers?
  2. If I have 2 processes running, how would I ensure this activity will be run on both processes?

I couldn’t find reference in Java API doc for DescribeTaskQueue and am confused about the registration part.

Appreciate any help on this topic.

4

Is it because a workflow can get picked up by a worker running in a different process/pod?

In case of a worker failure (for example it goes down) your workflow execution could be continued on a different worker.

I couldn’t find reference in Java API doc for DescribeTaskQueue and am confused about the registration part.

Here is an example:

TaskQueue tq = TaskQueue.newBuilder().setKind(TaskQueueKind.TASK_QUEUE_KIND_NORMAL)
   .setName(taskQueueName).build();

        DescribeTaskQueueRequest req = DescribeTaskQueueRequest.newBuilder()
                .setNamespace(client.getOptions().getNamespace())
                .setTaskQueue(tq)
                .setTaskQueueType(TaskQueueType.TASK_QUEUE_TYPE_WORKFLOW)
                .build();
        DescribeTaskQueueResponse res = service.blockingStub().describeTaskQueue(req);
        for (PollerInfo pollerInfo : res.getPollersList()) {
           // get poller identity from pollerInfo.getIdentity();
        }

You can set identity in your client via WorkflowClientOptions (identity is used to identify a worker and is also recorded in workflow history), for example:

    WorkflowClient client = WorkflowClient.newInstance(service,
            WorkflowClientOptions.newBuilder()
                    .setIdentity("myIdentity")
                    .build();
5
  1. how would this activity find the workers?

One way described is using DescribeTaskQueue api

  1. If I have 2 processes running, how would I ensure this activity will be run on both processes?

I believe in this case each process would have a worker polling that worker-specific task queue that hosts this activity.