Possibility of scalable buffered counter

Community Support
,
1

Hey , I am Junaid , I am new to temporal, it would be really helpful if some could help on this.

  • I am thinking we can use temporal to implement scalable buffered counter ( very frequent 1/s to 100/s update ) like work flow, where we can make a blocking work flow (do we really need to block)? , and increment the counter using signals and get the the count by querying , what is the possibility of this ?.

  • Can we make a workflow where it periodically do some tasks , update something in every 1 sec ? (can we return error forever to do this task or can we use ContinueAsNew for this )?.

  • How to use workflow starter client service ( i understood how to create worker and workflows and activities) can we use simple apis to trigger this (workflow starting) ? how it will be in go ?. I know we can do this for signalling by simple api call.

thanks in advance ,
Junaid

1 Like
2

Welcome, Junaid!

Counter

Temporal is not a good fit for a very frequently updated counter. The temporal core abstraction is a workflow. And each workflow execution (aka instance) has limited throughput at this point. Each update to a single workflow execution requires a database write, so the number of updates to a single execution is limited by the roundtrip DB latency. So counters which go above ten updates per second would have contention as only one update at a time is allowed for the given workflowId.

Temporal Scales very well with the number of parallel workflow executions. Use cases that require not a very high rate of update to a single workflow execution, but have a high aggregated number of updates work very well on top of it. For example, a customer loyalty program with hundreds of millions of customers and IoT device management application with workflow execution per device are some of the successful production use cases.

In your case, you still could use Temporal if you partition your counter to multiple workflow executions. For example, if the updates are evenly distributed (using some hash function) across 10 workflows you would be able to support your rate without a problem. Then to get a count you would need to query all 10 of them.

If you need a highly scalable eventually converging counter, then you can create a hierarchy of workflows.
For example a top level workflow starts 100 children, each of them starting 100 children. The leaf children will get increment requests in a form of signals and signal their parent periodically (let’s say once a minute). This way all the counts are going to converge at the root parent within a couple of minutes. If such a counter should be long lived the design will be complicated by the need to call continue as new periodically to reset history sizes back to 0. Let me know if this is something of interest and I can help with the details.

Periodic Tasks

The design depends on how frequently you want to execute the task.

  • For infrequent executions (every 10 seconds or slower) use the server-side retry. Specify a RetryPolicy (or RetryOptions in Java) when invoking the activity. In the RetryPolicy specify an exponential coefficient of 1 and an initial interval of the retry frequency. Then fail the activity to retry it up to the specified retry policy expiration interval.
  • For very frequent executions of every few seconds or faster the solution is to implement the logic inside an activity implementation as a loop that calls whatever logic needed and then sleeps for the retry interval. To ensure that the periodic activity is restarted in a timely manner in case of worker failure/restart the activity has to heartbeat on every iteration. Use an appropriate RetryPolicy for the restarts of such failed activity.
  • In a rare case when periodic execution of a sequence of activities is needed or activity arguments need to change between retries a child workflow can be used. The trick is, that a parent is not aware of a child calling continue as new. It only gets notified when a child completes or fails. So if a child executes the sequence of activities in a loop and calls continue as new periodically the parent is not affected until the child completes.

Starting Workflows

Temporal service exposes gRPC API for all its functionality. Starting workflow is done through StartWorkflowExecution or SignalWithStartWorkflowExecution. Both Go and Java provide higher-level SDKs that wrap gRPC API in a more developer friendly layer. You can call these APIs from any process or if you have a need to call them from not currently supported languages you can use gRPC generated bindings directly.

Here is a Go Sample of starting a workflow (with error handling and cleanup logic omitted):

// The client is a heavyweight object that should be created once per process.
	c, _ := client.NewClient(client.Options{
		HostPort: serviceHostPort,
	})
	workflowOptions := client.StartWorkflowOptions{
		ID:        workflowID,
		TaskQueue: taskQueue,
	}
	workflowRun, _ := c.ExecuteWorkflow(context.Background(), workflowOptions, child_workflow.SampleParentWorkflow)
	fmt.Printf("Started workflow with WorkflowID=%s and RunID=%s", workflowRun.GetRunID(), workflowRun.GetRunID())

Here is the full Go sample I took this code from.

2 Likes
4

thank you @Maxim for the detailed explanation