Experiencing sudden increase in Postgresql CPU consumption

[Anunay_Biswas]

I am running temporal with Postgresql 11. Postgresql server experiences a sudden increase in CPU consumption whenever there is a slight jump in requests per minute. Can you please guide me how can I debug this ? Postgresql has 8 cores. There are 8 history pods with 1 core / 1 GB each. 2 frontend and 2 matching pods with same resources.

Each request starts a new workflow or issues a signal if workflow is already running.

1. Workflow waits for few signals
2. Workflow contains 10-30 Activity calls.
3. Several activities which typically take less than 1 seconds are local activities. Rest of the activities finish in less than 5 seconds.

[tihomir]

Postgresql server experiences a sudden increase in CPU consumption whenever there is a slight jump in requests per minute

Can you monitor two things (server metrics) when you increase your rps:

Sync match rate (should be > 95% preferably > 99%, if lower or decreases would mean you most likely need to increase your worker capacity):
sum(rate(poll_success_sync{}[1m])) / sum(rate(poll_success{}[1m]))

Number of persistence requests (could also indicate underprovisioned workers), not sure if this is exact query you show in the second picture:
sum(rate(persistence_requests{operation="CreateTask"}[1m]))