At the beginning of this week a colleague of mine said to me that some JDBC connections were left open in one of our web projects. Before continuing to the rest of the story, let me first draw a rough architectural picture of the project.
We use JPA/Hibernate and Spring transactions decleratively in data access layer. In presentation layer, we use Spring WebFlow 1.0.x and keep EntityManager objects open as long as current flow is active to get rid of infamous lazy exception problems.
After all, as we don’t directly deal with JDBC connections in any other point, and have a very standard configuration of Spring with JPA/Hibernate, at first I was sceptical about his observation. We keep EntityManager objects open as long as flows are active, whose duration span user think time. First, I thought that connections are kept open as long as JPA EntityManager objects are open, and there might have been a problem in closing those EntityManager objects in our codebase. However, we quickly concluded that we were closing
EntityManager objects properly, and apart from that, we found out that EntityManager and JDBC Connection open/close operations are not related with each other. EntityManager opens a JDBC connection when it is really needed. Specifically, it opens a new connection when a new transaction begins and closes it at the end of the transaction.
I then said to him that it might most probably be a bug in Hibernate, and suggested to upgrade Hibernate version from 3.2.6 to 3.3.1. Unfortunately, upgrade to latest Hibernate revision didn’t help.
After one or two days of debugging he popped back to me and said that he had found two places which were guilty of those open connections. One place was in Hibernate code. We are using dbtimestamp type which maps to database’s current timestamp, rather than JVM’s current timestamp. In DbTimestampType class, Hibernate uses a PreparedStatement to fetch db’s current timestamp. PreparedStatement is closed and its reference kept inside Batcher object is removed at the end of this operation. However, not closing Resultset retrieved while executing the PreparedStatement causes Hibernate Batcher object to keep reference to the Resultset object. As a result Batcher object doesn’t close JDBC connection as it still has a reference to that Resultset.
There is also an open issue in Hibernate JIRA which is closely related with our case. Instead of waiting for the next release of Hibernate for the solution, we decided to copy DbTimestampType’s source code, fix the problem, and create our own user type namely dbtimestamp2 instead of Hibernate’s registered dbtimestamp type.
The second place which was under suspicion as being cause of those open connections was Spring’s HibernateJpaDialect class. Spring’s JpaTransactionManager exposes JpaTransactionObject as JdbcTransactionObject when DataSource object in
JpaTransactionManager is not null. This might occur in cases when JPA is configured in standalone mode or a DataSource object is injected directly into the JpaTransactionManager. By that way, plain JDBC operations are able to share same transaction context as with JPA operations. JpaTransactionManager does this by first getting Connection object through calling configured JpaDialect’s getJdbcConnection() method, and then bounding DataSource object together with this connection in the current thread context.
If we examine HibernateJpaDialect.getJdbcConnection() method closely, we see that Connection is retrieved by calling Session.connection() method. Session.connection() method returns a borrowed connection which is actually a proxy object to java.sql.Connection. Application code accesses actual JDBC connection through the borrowed connection. Hibernate’s ConnectionManager manages borrowed connection by keeping a reference to it when it is instantiated, and removes the reference when its private cleanup() method is called. cleanup() method actually calls ConnectionManager.releaseBorrowedConnection() method to release it. Another case which triggers borrowed connection release is calling close() method of borrowed connection itself. The conditions at which cleanup() is called method are listed listed in
Hibernate source code as follows;
- At the end of the Session
- At manual disconnect of the Session
- From afterTransaction(), in the case of skipped aggressive releasing
When we look at those three locations, we see that it is called at first two points, however there is no cleanup() call in ConnectionManager.afterTransaction() method. Calling afterTransaction() method causes physical JDBC connection to be closed, but as cleanup() method is not called, borrowed connection is not released after transaction completion.
The problem actually begins after transaction completions. When we trigger a JDBC statement to be executed without a transaction, for example, by accessing a lazy collection’s elements, Hibernate opens a new JDBC connection to fetch collection elements from DB. If connection release mode of Hibernate is set to AFTER_STATEMENT or AFTER_TRANSACTION (provided there is no active TX at that point), then Hibernate attempts to close JDBC connection by calling afterStatement() method. Unfortunately, connection cannot be closed, if borrowedConnection is not null, in other words there is a reference to a borrowed connection object. As you remember this reference is left after previously executed transaction. As a result, the physical JDBC connection will be kept open until a new transaction is initiated or Session is closed, or it is manually disconnected. In our case if user stories have many non transactional data access, or they require long user think time, keeping EntityManager/Session open until the end of flow, might cause connections to be exhausted.
JpaTransactionManager does a resource clean up after transaction completion, and calls JpaDialect’s releaseJdbcConnection() method. However, HibernateJpaDialect does nothing in its releaseJdbcConnection(). If it were called previouly retrieved connection’s close() method, it would cause release of borrowed connection, because as I said before the connection returned from Session was actually borrowed connection, which is a Connection proxy, calling its close method triggers ConnectionManager’s releaseBorrowedConnection.