Batching requests to Elasticsearch

RyanThomas · October 19, 2020, 8:10pm

When Temporal is handling a high volume of workflows per second, does batch together documents for multiple workflow executions into one request to Elasticsearch?

Wenquan_Xing · October 19, 2020, 8:21pm

Yes:

github.com

temporalio/temporal/blob/e299cdd3dae36776312c656c5d34602fdf3f9bc2/service/worker/indexer/esProcessor.go#L102-LL111


	params := &es.BulkProcessorParameters{
		Name:          processorName,
		NumOfWorkers:  config.ESProcessorNumOfWorkers(),
		BulkActions:   config.ESProcessorBulkActions(),
		BulkSize:      config.ESProcessorBulkSize(),
		FlushInterval: config.ESProcessorFlushInterval(),
		Backoff:       elastic.NewExponentialBackoff(esProcessorInitialRetryInterval, esProcessorMaxRetryInterval),
		BeforeFunc:    p.bulkBeforeAction,
		AfterFunc:     p.bulkAfterAction,
	}

Topic		Replies	Views
Does temporal also emit activity related events to ES? Community Support elasticsearch	3	476	February 24, 2022
Temporal 1.9.2 not working with Elasticsearch 7.7 - Error 400 All Shards Failed Community Support elasticsearch	3	2544	July 21, 2021
Indexing workflows for search / filtering Community Support	2	1027	August 18, 2020
Timeouts on visibility-tasks but workflows still completing Community Support	3	371	January 11, 2023
Elasticsearch for visibility store Community Support elasticsearch	0	241	October 24, 2023

Batching requests to Elasticsearch

Related Topics