Event-store vs. Visibility-store -- questions

Support
#1

Hi, could someone please provide a concise description of event-store vs. visibility-store, to include relationship to ElasticSearch. I’m familiar with how to query the event-store, but the visibility-store (the description of) is . . . opaque. For example, you can attach search-attributes to the workflow, but are they only queryable from ElasticSearch? Also, within a workflow, how does one attach visibility meta to the workflow execution?

Most importantly: if you’re adding visibility meta to the visibility-store, what actually gets written? Does it include the event data? Is there a mechanism to build aggregations of meta entries? Imagine you have a workflow with 10,000 parts, where each part represents an S3 multi-part upload. Going further, you could attach heartbeat (meta?) to each activity as it progresses, updating # bytes uploaded. Displayed on Temporal UI, it’s too much information. We’ll likely build a separate UI (which is fine). But, you need the ability to reduce/aggregate. Is this something the visibility “engine” supports? Or, is the idea that you build your own event-stream out of Temporal, and aggregate that way?

Many thanks!

#2

Event Store

Event store persists workflow execution histories. Its primary key is Namespace, WorkflowId, RunId. The API to retrieve a history is GetWorkflowExecutionHistory. Both UI and tctl use this API to show the history events.

Visibility Store

The visibility store persists so-called visibility records. Each visibility record contains information about a single workflow. It is possible to attach an application-specific information as memo field when starting a workflow. It is also possible to specify custom search attributes when starting a workflow and insert/update them when executing a workflow. See documentation for details.
The ListOpenWorkflowExecutions, ListClosedWorkflowExecutions and ListWorkflowExecutions are used to retrieve visibility records. The only aggregation which is supported (when running with ES) is count.

Questions

if you’re adding visibility meta to the visibility-store, what actually gets written?

I’m not sure what do you mean by “visibility meta”.

Does it include the event data?

No, event data is stored in the Event Store.

Is there a mechanism to build aggregations of meta entries?

Currently, no such mechanism exists. We have some ideas like supporting CRDT and hierarchical aggregation, but nothing concrete.

Is this something the visibility “engine” supports? Or, is the idea that you build your own event-stream out of Temporal, and aggregate that way?

Visibility only supports simple list and count queries. At this point, you would need your own solution to support such aggregation requirements.