We run a multi-tenant SaaS on Temporal and are trying to answer a simple operational question: which tenants are generating the workflow and activity load at a given moment? We would like to know what the recommended approach is.
Setup
- Self-hosted Temporal Server 1.25.2 (planning an upgrade to 1.31+)
- TypeScript SDK 1.11.7
- A few thousand tenants sharing the same namespaces and task queues. The tenant id is part of every workflow id, e.g.
- SDK and server metrics are scraped via Prometheus into our observability backend
What we would like to see
- Activity executions (and ideally latency and failures) per tenant over time
- During a burst or a backlog, which tenants are behind it
What we have considered
- Custom SDK metrics (metricMeter, TS SDK ≥ 1.12) with a tenant tag. This is the closest fit, but with thousands of tenants × activity types the series count is large, and on a Prometheus scrape a single scrape over the sample limit drops every series of the worker. Counter-only, or exporting over OTLP instead of the scrape endpoint, would help — but is a tenant tag on SDK metrics something you would recommend at all?
- An activity interceptor that writes one log line per execution with tenant, activity type, duration and outcome, then aggregating in the log backend. Works with our current versions and avoids metric cardinality, at the cost of log volume.
- A custom search attribute TenantId set at workflow start. It covers workflows but not activities, and count … GROUP BY only supports ExecutionStatus, so “top tenants by running workflows” still needs per-tenant queries or client-side aggregation.
- Fairness keys (server 1.31 GA) keyed by tenant. This addresses the root problem — one tenant’s burst delaying everyone — but as far as we can tell it does not expose per-key metrics.
Questions
- What do other multi-tenant users do for per-tenant visibility of workflows and activities? Is one of the approaches above the recommended one, or is there something we are missing?
- If a tenant tag on SDK custom metrics is acceptable, what are the guidelines for cardinality (counters only, OTLP export, bucketing tenants)?
- With fairness enabled, is there any per-fairness-key visibility (backlog, dispatch rate), or is that intentionally left out because of cardinality?
- Since we are upgrading anyway: is there anything in recent server releases (1.26 → 1.32), or coming soon, that covers this requirement and that we may have missed in the release notes? For example per-key task queue stats, visibility aggregations by search attribute, or new metric labels.
Thanks!