How to schedule workflow based on different types of constraints

Community Support
1

How do I submit a workflow so it will be scheduled to a proper worker?

  • hard constraints - CPU and GPU of the worker must match that specified to the workflow
  • priority - higher priority workflows are scheduled first
  • soft constraints - a worker with a matching value, if existed, will be scheduled first. Otherwise, any worker matches the hard constraints can.

Thank you

2

In Temporal, we don’t schedule workflows to specific workers. We do it with activities as they are actual consumers of CPUs/GPUs and other resources.

  • hard constraints - CPU and GPU of the worker must match that specified to the workflow

You use a separate activity task queue per GPU type.

  • priority - higher priority workflows are scheduled first

Temporal doesn’t support priorities natively yet. Depending on your use case, different workarounds are possible. What is the maximum rate of activities per second you need to schedule across all priorities?

  • soft constraints - a worker with a matching value, if existed, will be scheduled first. Otherwise, any worker matches the hard constraints can.

Depending on the rate of execution different solutions are possible.

3

Hard constraints:

You use a separate activity task queue per GPU type.

We will have to literally create ten of thousands of queues for each combinations of the hard constraints. Some constrains are a list, for example a list of OSes and a list of test programs.

The rate of execution is roughly 500K jobs per day on 11K machines. Tyically the submission rate is a few jobs per seconds at the peak hours. Thanks

4

How can I set constraints on “Activities” then so certain workers with specific attributes can run activities with matching attributes?

Hard constraints, soft constraints, and priority, anything possible on activities?

Thanks

5

We will have to literally create ten of thousands of queues for each combinations of the hard constraints. Some constrains are a list, for example a list of OSes and a list of test programs.

I see. In this case, I would move the constraint matching outside of Temporal. Implement an activity that given constraints returns the hostname to run it on. Then use a host specific task queue to run the actual computing activity at that host.

6

what features does Temporal have for filtering/scheduling “activities”? where can I find the document? Thanks

7

The only “filtering” is on the level of task queue. All activities scheduled to a given task queue will be delivered to the workers that listen on it without any filtering.