Batching requests to Elasticsearch

RyanThomas · October 19, 2020, 8:10pm

When Temporal is handling a high volume of workflows per second, does batch together documents for multiple workflow executions into one request to Elasticsearch?

Wenquan_Xing · October 19, 2020, 8:21pm

Yes:

github.com

temporalio/temporal/blob/e299cdd3dae36776312c656c5d34602fdf3f9bc2/service/worker/indexer/esProcessor.go#L102-LL111


	params := &es.BulkProcessorParameters{
		Name:          processorName,
		NumOfWorkers:  config.ESProcessorNumOfWorkers(),
		BulkActions:   config.ESProcessorBulkActions(),
		BulkSize:      config.ESProcessorBulkSize(),
		FlushInterval: config.ESProcessorFlushInterval(),
		Backoff:       elastic.NewExponentialBackoff(esProcessorInitialRetryInterval, esProcessorMaxRetryInterval),
		BeforeFunc:    p.bulkBeforeAction,
		AfterFunc:     p.bulkAfterAction,
	}

Topic		Replies	Views
Action batching Community Support	3	1017	August 23, 2022
Will temporal fit my scenario Community Support	7	70	February 5, 2025
Does temporal also emit activity related events to ES? Community Support elasticsearch	3	530	February 24, 2022
Temporal performance with golang microservice, Cassandra & Elasticsearch Community Support go-sdk , elasticsearch , cassandra , docker , performance	14	3418	February 1, 2023
DB migration support for Temporal deployment Server Deployment elasticsearch , cassandra , postgresql	4	1190	April 1, 2023

Batching requests to Elasticsearch

Related topics