@maxim@tihomir We’re seeing this as well causing problems with cloud-sql=proxy IAM integration when we run more than 3 history nodes. How do we configure Temporal to keep connections alive longer and start new connections less often?
I’ve added a screenshot where you can see that Temporal is opening a whopping 250 new connections per second which I assume is unintentional.
We are observing similar pattern as well, one of the solution is managing connection pool outside of temporal using proxy or pgbouncer ( we haven’t taken a decision thought).
is there a better/efficient solution to this ?, Also any insights on why temporal would spawn so many connections.
We’re doing exactly this, but under high strain things still fall apart for us with connections breaking at the temporal side. I am quite confident it has something todo with the custom connection pool and reference counting on the connections that is implemented in temporal fpr postgres.
Recently we experienced an outage we were not able to fix besides the, let’s prevent more activities to be created and let’s drain the existing work queues.
is there any property called “keepAlive” we can use ?
Temporal static config allows you to set maxConnLifetime, which you set to 30m (default is 1hr) which sets the connection lifetime duration.
Setting this to 0 would mean on postgres side you would want to reuse the conn forever.
Would also look in your postgres config to see if you are setting idle_in_transaction_session_timeout and if so try to set it to 0 (to disable it) or a larger duration
