# Auto scaling worker deployment **URL:** \ **Category:** Community Support\ **Tags:** python-sdk, scaling, deployment\ **Created:** [March 28, 2024, 5:02pm UTC](https://community.temporal.io/t/auto-scaling-worker-deployment/11547 "2024-03-28T17:02:33Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![mmoya](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/mmoya/32/6833_2.png) [@mmoya](https://community.temporal.io/u/mmoya)\ **Post date:** [March 28, 2024, 5:02pm UTC](https://community.temporal.io/t/auto-scaling-worker-deployment/11547/1 "2024-03-28T17:02:33Z")
Hello, I created a chart for deploying a temporal worker using [this](https://keithtenzer.com/temporal/Deploying_Temporal_Worker_on_Kubernetes/) guide and I’m currently considering a HPA on that deployment and defining ```python my_worker = Worker( client, task_queue="some-task-queue-str", activities=["some-activity-defn"], max_concurrent_activities=5, ) ``` via the application’s worker However, as far as the deployment itself, the HPA is currently only scaling on memory (or CPU utilization) so it looks something like ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: some-temporal-name spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: some-temporal-name minReplicas: 1 maxReplicas: 5 metrics: {{- if .Values.autoscaling.CPUUtilization }} - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 {{- end }} {{- if .Values.autoscaling.MemoryUtilization }} - type: Resource resource: name: memory target: type: Utilization averageUtilization: 70 {{- end }} ``` are there any other recommended metrics to consider for autoscaling our activities besides the pod’s memory usage? ---
**Author:** ![Chad\_Retz](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chad_retz/32/1396_2.png) [@Chad\_Retz](https://community.temporal.io/u/Chad_Retz)\ **Post date:** [March 29, 2024, 12:32pm UTC](https://community.temporal.io/t/auto-scaling-worker-deployment/11547/2 "2024-03-29T12:32:41Z")
Many people use “available slots” to know when to scale workers. See [Developer's guide - Worker performance | Temporal Documentation](https://docs.temporal.io/dev-guide/worker-performance). You would have benchmarked your workers and set their max-concurrent-activities set to a number you know the resources can handle, then you would watch if available slots get too low. ---
**Author:** ![mmoya](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/mmoya/32/6833_2.png) [@mmoya](https://community.temporal.io/u/mmoya)\ **Post date:** [March 31, 2024, 6:13pm UTC](https://community.temporal.io/t/auto-scaling-worker-deployment/11547/3 "2024-03-31T18:13:57Z")
thank you for the reply! That sounds great! As far as the python sdk for temporal, how do I evoke metrics like `temporal_worker_task_slots_available` or other `temporal_` metrics in my activity/worker? Is there a python example of this? are those default metrics that arise anytime I consider a metrics runtime(in this case, otel) like ```auto from temporalio.contrib.opentelemetry import TracingInterceptor from temporalio.runtime import OpenTelemetryConfig, Runtime, TelemetryConfig runtime = init_runtime_with_telemetry() ``` as far as my client ```auto # Connect client client = await Client.connect( client, # Use OpenTelemetry interceptor interceptors=[TracingInterceptor()], runtime=runtime, ) ``` or are these metrics that need to be defined in some way? ---
**Author:** ![Chad\_Retz](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chad_retz/32/1396_2.png) [@Chad\_Retz](https://community.temporal.io/u/Chad_Retz)\ **Post date:** [April 1, 2024, 1:31pm UTC](https://community.temporal.io/t/auto-scaling-worker-deployment/11547/4 "2024-04-01T13:31:37Z")
> [@mmoya](#): > > are those default metrics that arise anytime I consider a metrics runtime(in this case, otel) like Yes. [Here](https://github.com/temporalio/samples-python/blob/4303a9b15f4ddc4cd770bc0ba33afef90a25d3ae/open_telemetry/worker.py#L43-L48) is a sample of configuring OTel metrics on the runtime (which you only need to create one of and use when you’re creating your client). The interceptor is for tracing by the way, only the runtime needs to be configured for metrics. ---
**Author:** ![mmoya](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/mmoya/32/6833_2.png) [@mmoya](https://community.temporal.io/u/mmoya)\ **Post date:** [April 3, 2024, 6:56pm UTC](https://community.temporal.io/t/auto-scaling-worker-deployment/11547/5 "2024-04-03T18:56:37Z")
awesome, so I tried configuring our otel collector and our temporal client to consider the otel runtime but I’m unable to get the `temporal_` metrics at least for my remote worker. Is there anything else I would need to consider for a remote worker/activity? I also noticed, for the activites that were not remote, it was writing `temporal_` metrics to a service name called `temporal-core-sdk` despite defining a different service name as far as the provider [samples-python/open\_telemetry/worker.py at 4303a9b15f4ddc4cd770bc0ba33afef90a25d3ae · temporalio/samples-python · GitHub](https://github.com/temporalio/samples-python/blob/4303a9b15f4ddc4cd770bc0ba33afef90a25d3ae/open_telemetry/worker.py#L38) ---
**Author:** ![Chad\_Retz](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chad_retz/32/1396_2.png) [@Chad\_Retz](https://community.temporal.io/u/Chad_Retz)\ **Post date:** [April 4, 2024, 12:25pm UTC](https://community.temporal.io/t/auto-scaling-worker-deployment/11547/6 "2024-04-04T12:25:31Z")
> [@mmoya](#): > > Is there anything else I would need to consider for a remote worker/activity? No, this should work without issue if you give it an OTLP gRPC endpoint. > [@mmoya](#): > > it was writing `temporal_` metrics to a service name called `temporal-core-sdk` despite defining a different service name as far as the provider [samples-python/open\_telemetry/worker.py at 4303a9b15f4ddc4cd770bc0ba33afef90a25d3ae · temporalio/samples-python · GitHub](https://github.com/temporalio/samples-python/blob/4303a9b15f4ddc4cd770bc0ba33afef90a25d3ae/open_telemetry/worker.py#L38) That link is to a trace provider and is unrelated to metrics. When configuring `TelemetryConfig` for the metrics, you can set `attach_service_name` to overwrite the default of `temporal-core-sdk`. ---
**Author:** ![mmoya](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/mmoya/32/6833_2.png) [@mmoya](https://community.temporal.io/u/mmoya)\ **Post date:** [April 4, 2024, 2:18pm UTC](https://community.temporal.io/t/auto-scaling-worker-deployment/11547/7 "2024-04-04T14:18:45Z")
> [@Chad\_Retz](#): > > attach\_service\_name @Chad_Retz thank you for the reply! I’m a little stumped as to why I’m able to send custom metrics yet unable to send the default `temporal_` metrics to our otel instrumentator+dd via our remote worker. Would you suggest revisiting the configs? As far as `attach_service_name` would setting it to `True` overwrite the service\_name from `temporal-core-sdk` to the service name we provide? Or would we need to set it to `False`? (I noticed it seems to be set to `True` by default [sdk-python/temporalio/runtime.py at main · temporalio/sdk-python · GitHub](https://github.com/temporalio/sdk-python/blob/main/temporalio/runtime.py#L338)) ---
**Author:** ![Chad\_Retz](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chad_retz/32/1396_2.png) [@Chad\_Retz](https://community.temporal.io/u/Chad_Retz)\ **Post date:** [April 4, 2024, 3:09pm UTC](https://community.temporal.io/t/auto-scaling-worker-deployment/11547/8 "2024-04-04T15:09:41Z")
> [@mmoya](#): > > Would you suggest revisiting the configs? Yes, make sure that the endpoint you are giving as `metrics=OpenTelemetryConfig(url="http://whatever")` accepts OTel metrics, and that you are creating that one global runtime and using it across all clients you create. > [@mmoya](#): > > As far as `attach_service_name` would setting it to `True` overwrite the service\_name from `temporal-core-sdk` to the service name we provide? It is defaulted to `True` which means the SDK attaches the service name. You set it to `False` to stop setting it. You can set `global_tags` to set any tags for all metrics (including `service_name` after setting attach to false). ---
**Author:** ![mmoya](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/mmoya/32/6833_2.png) [@mmoya](https://community.temporal.io/u/mmoya)\ **Post date:** [April 4, 2024, 3:58pm UTC](https://community.temporal.io/t/auto-scaling-worker-deployment/11547/9 "2024-04-04T15:58:00Z")
> Yes, make sure that the endpoint you are giving as `metrics=OpenTelemetryConfig(url="http://whatever")` accepts OTel metrics, and that you are creating that one global runtime and using it across all clients you create. Thank you for the reply. I’m considering a global runtime and I’m able to write custom metrics using that endpoint(our endpoint is `http://localhost:4317` since our app/worker also contains a otel sidecar upon deployment). It’s just not writing the _default_ metrics. As a workaround, is there a way to import/consider default metrics as a variable so that I can manually add it as a metric in otel as I would a custom metric? Would love to be able to import `temporal_worker_task_slots_available` in some way and manually record that metric into otel. Or I was wondering if there was source code where `temporal_worker_task_slots_available` is computed? > It is defaulted to `True` which means the SDK attaches the service name. You set it to `False` to stop setting it. You can set `global_tags` to set any tags for all metrics (including `service_name` after setting attach to false). Just to triple check I’m understanding this correctly, if I set to `attach_service_name=False` ```python Runtime( telemetry=TelemetryConfig( metrics=OpenTelemetryConfig(url="http://localhost:4317"), attach_service_name=False, global_tags={"service_name": "my-service"} ) ) ``` then it would use the same service\_name as the service name for the provider ```python provider = TracerProvider(resource=Resource.create({SERVICE_NAME: "my-service"})) ``` in this case `"my-service"`? Thus, if we set `attach_service_name=False` those metrics would now be under `service_name:my-service` instead of `service_name:temporal-core-sdk`? ---
**Author:** ![Chad\_Retz](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chad_retz/32/1396_2.png) [@Chad\_Retz](https://community.temporal.io/u/Chad_Retz)\ **Post date:** [April 4, 2024, 10:39pm UTC](https://community.temporal.io/t/auto-scaling-worker-deployment/11547/10 "2024-04-04T22:39:57Z")
> [@mmoya](#): > > able to write custom metrics using that endpoint Can you clarify “write custom metrics”? Is this using `my_runtime.metric_meter().[whatever]` or some other tool? Does directly making metrics on the meter work? Are you sure you are setting this runtime as the client options? > [@mmoya](#): > > Or I was wondering if there was source code where `temporal_worker_task_slots_available` is computed? This is deep in the Rust core and consuming the metric via the metric system may be the best way. Python does have the ability to now use a metric buffer where you can manually consume metrics instead of exposing via otel/prometheus. We don’t have a sample yet, but see [this test](https://github.com/temporalio/sdk-python/blob/b07e75ee124c7d5e7bf1fc5e587963549189cca6/tests/worker/test_workflow.py#L3550). > [@mmoya](#): > > then it would use the same service\_name as the service name for the provider This is confusing “tracing” and “metrics”. `global_tags` applies to metrics and is unrelated to `TracerProvider`, and yes if you set it with `service_name` tag it should be on every metric.