Not sure if this is normal. We have each service running in its own container. History seems to be the only one doing any real work. The one with SERVICES=worker set is basically idle, and has this in it’s logs:
I would have thought “worker” is the service that does the real work, so kinda confused.
Understand, and yes the “worker service” name is confusing. Maybe think of it as the “system workflows service”. It runs background system workflows for things like scavengers as you noticed but also batch operations, archival, namespace retention, etc.
I guess my question is, when someone dispatches a “real” task via an SDK, should it be running on history
From a high level view, frontend, history and matching services are the ones that are involved with your workflow executions directly. Frontend service you can think of as a “dispatcher”, all client calls go through it and it then forwards them. History service is the one that hosts your shards. Each shard hosts one or more of your workflow executions, meaning it is responsible to create and maintain the execution event history (and persist it), mutable state (server view of the exec), and has transfer queues responsible to ship different events to other services such as matching (for workflow, activity tasks, timers etc) as well as visibility (moving visibility tasks to your visibility store).
Matching service receives workflow/activity tasks for history service and it hosts the task queues that your workers define to poll your workflow and activity tasks from (your workers actually poll frontend service, but then frontend forwards these poll requests to matching).