Hello. I currently have a deployment in k8s via de Temporal Helm repository with 1.8.0 version.
For persistence I use a managed Postgres from Azure. This is a MVP project.
I have two environments with isolated resources.
In preprod I have 2Vcpu and in prod 4Vcpu.
The temporal runs a distributed cron job at 10 minutes interval, whose logic is implemented by a backend.
Due to the fact I noticed some high cpu consumption in preprod I stopped the backend process, leaving in preprod running only temporal.
At some point I started having almost contant >90% cpu consumption on preprod, while in prod I have constant ~10%.
After I restarted all pods from temporal seems CPU for now went down to around ~10 percent in preprod as well, or anyway very low comparing to the constant spike I had.
Postgres is version 11 in Azure.
And also data retention for namespace in 7 days.
Now my question is, what could have caused the high CPU, since even when only the temporal was running with a cron job activated, I still had same CPU, after stopping the backend.
For the cron job workflow I also have some retries set and timeouts on client side, but anyway the backend was stopped.
Do you have any insight I can look into. I know there were little resources in the envs, but it’s an MVP project, and what worried me is the difference between prod and preprod.