Running out of db connections

Community Support
1

I’ve converted some of my code over to temporal. I went from using JMS Listeners that processes each JMS message one at a time over to using temporal workflows.

But during performance testing I keep running out of db connections. I’ve tried configuring temporal to run each activity sequentially by configuring activiePoolSize/WorkflowPoolsize to 1 and even having a low max maxTaskQueueActivitiesPerSecond value. I am using Spring boot to manage my db connection pooling. Are the workers not releasing the db connections? I don’t have any issues when using JMS Listeners.

2

Are the workers not releasing the db connections?

Workers do not establish a db connection. Could you provide more info?
Are you creating your db connections in your workflow or activity code?
Can you show more info on that so we can understand what could be going on?

3

I am load testing microservice, specifically many endpoints that make db calls as well and some endpoints that asynchronously invoke temporal workflows which in turn run on the microservice and also make db calls.

I am not creating db connections in workflow or activity code. The activity calls spring @Repository services that are has which in turn uses myBatis to call the database. I am using HikariPool jdbc connection pool. The same pattern is used throughout the microservice.

Also using GitHub - applicaai/spring-boot-starter-temporal: The driver making it convenient to use Temporal with Spring Boot.

The logs reveal that it is not temporal threads that are running out of connections, its the http threads servicing the http requests.

j.s.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
	at c.z.h.p.HikariPool.createTimeoutException(HikariPool.java:695)
	at c.z.h.p.HikariPool.getConnection(HikariPool.java:197)
	at c.z.h.p.HikariPool.getConnection(HikariPool.java:162)
	at c.z.h.HikariDataSource.getConnection(HikariDataSource.java:128)
	at o.s.j.d.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:265)
4

Don’t think we can give specific advice for GitHub - applicaai/spring-boot-starter-temporal: The driver making it convenient to use Temporal with Spring Boot.
as it’s not a project maintained by Temporal.

That said, do you make sure that you close your connections in activities? Remember that you register a single activity implementation with the worker, and i assume your activities are singletons (@Component) in that spring boot starter framework? Might be worth making sure that you close connections in code explicitly before activity method completes.

5

Dear carrotcap3… Did you ever resolve this issue? Can you please share what you found out?

6

the issue was not with temporal, temporal only increased throughput leading to running out of db connections.