Redistribute pending activities among new workers after scaling up

Community Support
,
1

I have 3 workers by default which are polling for an activity on a task-queue.
If I schedule N number of activities on the task-queue, all N activities are being assigned among these 3 workers.

I want the ability to scale the workers up, so that the pending activities are redistributed among the new workers.

Is this possible? Currently, even on bringing up new workers, the existing activities stay in pending state and wait for existing workers to free up.

2

the existing activities stay in pending state

whats the pending state details, as in does it stay activity task scheduled or started?
if started, then activity task was already dispatched to your activity worker.

can you disable eager activities in your worker config, for example in java sdk set
WorkerOptions->disableEagerExecution to true, and see if that gives you better activity distribution across newly added activity workers.

3

whats the pending state details

It is in ActivityTaskScheduled state.
But I noticed that the dashboard doesnt reflect ActivityTaskStarted event until after the activity is finished.

Edit: Even with WorkerOptions->disableEagerExecution set to true, the new workers are sitting idle while there are scheduled activities in the queue.

4

But I noticed that the dashboard doesnt reflect ActivityTaskStarted event until after the activity is finished.

Correct, Temporal server does not record each activity retry in event history. ActivityTaskStarted event is written when last attempt of activity completes.

the new workers are sitting idle while there are scheduled activities in the queue.

Are you sure your workers are polling for activity tasks and have those activity types registered? Do you have worker and server metrics configured so maybe we can take a closer look?

5

Yes the workers are polling for the activity tasks
The setup is following

  • There is a Java application which start an asynchronous child workflow which in turn starts an activity. The application does not wait for the activities to complete.
  • The application can start any number of such activities continuously.
  • The worker is a Python program, which implements the activity and polls on the task queue.
  • There are multiple such workers running.

Now when the number of activities in Scheduled state increase, I want to scale the workers. But the scaled workers pick only the newly created activities and don’t pick the already scheduled activities.

Currently we don’t have metrics configured. Will configure and attach the metrics as well.

6

Currently we don’t have metrics configured. Will configure and attach the metrics as well.

sounds good, lets continue here once we have this in place as think will help

The application does not wait for the activities to complete.

Do you mean parent workflow does not wait for child to complete right (as in start child in abandoned mode). Or do you really mean you start activity async and you complete the child right after?

and don’t pick the already scheduled activities.

For these “scheduled” ones is there a way to see their pending activity info either in ui or via temporal workflow describe cli command? As mentioned ActivityTaskScheduled event being last event in history does not mean the activity wasnt dispatched to your activity worker.