CPU consumption - Postgres

Community Support
1

Hello. I currently have a deployment in k8s via de Temporal Helm repository with 1.8.0 version.
For persistence I use a managed Postgres from Azure. This is a MVP project.

I have two environments with isolated resources.
preprod
prod

In preprod I have 2Vcpu and in prod 4Vcpu.

The temporal runs a distributed cron job at 10 minutes interval, whose logic is implemented by a backend.

Due to the fact I noticed some high cpu consumption in preprod I stopped the backend process, leaving in preprod running only temporal.

At some point I started having almost contant >90% cpu consumption on preprod, while in prod I have constant ~10%.

After I restarted all pods from temporal seems CPU for now went down to around ~10 percent in preprod as well, or anyway very low comparing to the constant spike I had.

Postgres is version 11 in Azure.
And also data retention for namespace in 7 days.

Now my question is, what could have caused the high CPU, since even when only the temporal was running with a cron job activated, I still had same CPU, after stopping the backend.

For the cron job workflow I also have some retries set and timeouts on client side, but anyway the backend was stopped.

Do you have any insight I can look into. I know there were little resources in the envs, but it’s an MVP project, and what worried me is the difference between prod and preprod.

2

I also want to mention that the high CPU started only after a very good number of days of continous run. So it might have so relevance to the issue.

3

can you provide the metrics below?

plz also include the operation information, e.g.
not sum(persistence_requests)
but sum(persistence_requests) by (operation)

persistence_requests
persistence_errors
persistence_latency