Experiencing sudden increase in Postgresql CPU consumption

Community Support
1

I am running temporal with Postgresql 11. Postgresql server experiences a sudden increase in CPU consumption whenever there is a slight jump in requests per minute. Can you please guide me how can I debug this ? Postgresql has 8 cores. There are 8 history pods with 1 core / 1 GB each. 2 frontend and 2 matching pods with same resources.

Screenshot 2022-09-13 at 10.52.34 PM
Screenshot 2022-09-13 at 10.52.34 PM3570×1110 185 KB

Screenshot 2022-09-13 at 10.52.55 PM
Screenshot 2022-09-13 at 10.52.55 PM2714×794 102 KB

Each request starts a new workflow or issues a signal if workflow is already running.

  1. Workflow waits for few signals
  2. Workflow contains 10-30 Activity calls.
  3. Several activities which typically take less than 1 seconds are local activities. Rest of the activities finish in less than 5 seconds.
3

Postgresql server experiences a sudden increase in CPU consumption whenever there is a slight jump in requests per minute

Can you monitor two things (server metrics) when you increase your rps:

Sync match rate (should be > 95% preferably > 99%, if lower or decreases would mean you most likely need to increase your worker capacity):
sum(rate(poll_success_sync{}[1m])) / sum(rate(poll_success{}[1m]))

Number of persistence requests (could also indicate underprovisioned workers), not sure if this is exact query you show in the second picture:
sum(rate(persistence_requests{operation="CreateTask"}[1m]))