Temporal not respecting concurrency configuration

Community Support
1

I am fairly new to Temporal, coming from Celery and Airflow type of things.

I am trying to execute a workflow with one activity. My use case is that only one workflow execution must run per worker. If there are 4 workers in the pool there should only be 4 active executions and others in pending/waiting state.

I have started the worker using following config

	w := worker.New(c, app.GreetingTaskQueue, worker.Options{
		MaxConcurrentActivityExecutionSize:     1,
		MaxConcurrentWorkflowTaskExecutionSize: 1,
	})

And I am submitting the workflow using following grpc request

    request = StartWorkflowExecutionRequest(
        namespace="default",
        workflow_id=workflow_id,
        workflow_type=WorkflowType(name=w_type),
        task_queue=TaskQueue(name=task_queue, kind=TaskQueueKind.TASK_QUEUE_KIND_UNSPECIFIED),
        input=DEFAULT_DATA_CONVERTER_INSTANCE.to_payloads(args),
        workflow_execution_timeout=timedelta(seconds=90),
        workflow_run_timeout=timedelta(seconds=90),
        workflow_task_timeout=timedelta(seconds=90),
        identity="",
        request_id="Do-not-know",
        retry_policy=RetryPolicy(
            initial_interval=timedelta(seconds=5),
            maximum_attempts=5
        ),
        workflow_id_reuse_policy=WorkflowIdReusePolicy.WORKFLOW_ID_REUSE_POLICY_UNSPECIFIED
    )
    workflow_svc = create_workflow_service("localhost", 7233, 10.0)
    response = await workflow_svc.start_workflow_execution(request=request)
    return response

Issue is when I spawn only 1 worker and fire 5 workflow executions using above code the worker starts executing all 5 workflow concurrently.

3

Rate limiting configs in worker.Options:

  • WorkerActivitiesPerSecond - worker specific rate limit for activities
  • MaxConcurrentActivityExecutionSize - worker specific limit on number of parallel activities
  • TaskQueueActivitiesPerSecond - global task queue activities per sec limit across all the workers

My use case is that only one workflow execution must run per worker

It’s not currently possible to rate limit concurrent workflow executions. For this use case you can have a single workflow you can send signals to. This workflow then can process signals in order (one at a time) and either execute the logic you need, or can invoke child workflow sync, assuring there is only one running at a time. When using child workflows you can reuse the parent workflow for different child types.

If there are 4 workers in the pool

Are all these workers listening on the same task queue in the same process? I think your don’t need more than one such worker in single process. For fault tolerance you should have workers (that listen to same task queue) in different processes like different hosts/availability zones.

And I am submitting the workflow using following grpc request

Any reason you are not using SDK api for this?

workflow_id=workflow_id,

I assume you are using a different workflow id for each execution. With Temporal you cannot have more than one workflow exec running with the same workflow id.

retry_policy=RetryPolicy(
initial_interval=timedelta(seconds=5),
maximum_attempts=5
),

Don’t think there is a need to define workflow retries (unless you really need it). Workflows do not have a default retry policy (unless you specifically set it). By default with Temporal unknown failures do now fail workflow execution but block it waiting for a fix. Activities on the other hand do have a default retry policy.

5

Hi @tihomir , could you point to the example needed to achieve this requirement? ( needed similar requirement and also facing same problem), or how to “assuring there is only one running at a time.” when using child workflow

6

Requirement

only one workflow execution must run per worker.

doesn’t make sense when applied to Temporal. Temporal ensures that a workflow keeps running in the presence of worker crashes. So if you tie a workflow to a worker, then the workflow will get stuck if the worker fails.

It is common to need to run a specific sequence of activities on a specific worker. Temporal supports this. See the fileprocessing sample.