[java] Oracle JDBC intermittent Connection Issue

I am experiencing a very strange problem This is a very simple use of JDBC connecting to an Oracle database

OS: Ubuntu
Java Version:  1.5.0_16-b02
               1.6.0_17-b04
Database: Oracle 11g Release 11.1.0.6.0

When I make use of the jar file OJDBC14.jar it connects to the database everytime When I make use of the jar file OJDBC5.jar it connects some times and other times it throws an error ( shown below) If I recompile with Java 6 and use OJDBC6.jar I get the same results as OJDBC5.jar

I need specific features in JODB5.jar that are not available in OJDBC14.jar

Any ideas

Error

> Connecting to oracle
    java.sql.SQLException: Io exception: Connection reset
    at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:74)
    at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:110)
    at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:171)
    at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:227)
    at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:494)
    at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:411)
    at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:490)
    at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:202)
    at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
    at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:474)
    at java.sql.DriverManager.getConnection(DriverManager.java:525)
    at java.sql.DriverManager.getConnection(DriverManager.java:171)
    at TestConnect.main(TestConnect.java:13)

Code

Below is the code I am using

import java.io.*;
import java.sql.*;
public class TestConnect {
    public static void main(String[] args) {
        try {
            System.out.println("Connecting to oracle"); 
            Connection con=null;
            Class.forName("oracle.jdbc.driver.OracleDriver");
            con=DriverManager.getConnection(
               "jdbc:oracle:thin:@172.16.48.100:1535:sample",
               "JOHN",
               "90009000");
            System.out.println("Connected to oracle"); 
            con.close();
            System.out.println("Goodbye");
        } catch(Exception e) { e.printStackTrace(); }
    }
}

OracleXETNSListener - this service has to be started if it was disabled.

run -> services.msc 

and look out for that services

Just to clarify - at least from what we found on our side! It is an issue with the setup of the randomizer for Linux in the JDK distribution - and we found it in Java6, not sure about Java7. The syntax for linux for the randomizer is file:///dev/urandom, but the entry in the file is (probably left/copied from Windows) as file:/dev/urandom. So then Java probably falls back on the default, which happens to be /dev/random. And which doesn't work on a headless machine!!!

-Djava.security.egd=file:/dev/./urandom should be right! not -Djava.security.egd=file:/dev/../dev/urandom or -Djava.security.egd=file:///dev/urandom

It's hard to say, but if I would check the actual version of the JDBC driver. Make sure it's 11.1.0.6.

Oracle doesn't include the database version in the filename. So the driver for 11.2 is the exact same name as the driver for 11.1 - ojdbc5.jar. I would extract the driver jar file, and find the MANIFEST.MF file, this will contain some version information. Make sure the version of the JDBC driver matches the version of your database. I suspect it may be a version issue, since there isn't a jar file named ojdbc14.jar on Oracle's 11.1.0.6 download page.

If the version matches - I'm out of ideas :)

I faced the same problem when a liquibase was executed from jenkins. Sporadically this error was thrown to the output and the liquibase change logs were not executed at all.

Solution provided: In the jenkin's maven project, the jdk was updated from jdk8-131 to any newer version (eg java8-162).

Disabling SQL Net Banners saved us.

The root cause of this problem has to do with user authentication versions. For each database user, multiple password verifiers are kept in the database. Typically when you upgrade your database, a new password verifier will be added to the list, a stronger one. The following query shows the password verifier versions that are available for each user. For example:

SQL> SELECT PASSWORD_VERSIONS FROM DBA_USERS WHERE USERNAME='SCOTT';

PASSWORD_VERSIONS
-----------------
11G 12C

When upgrading to a newer driver you can use a newer version of the verifier because the driver and server negotiate the strongest possible verifier to to be used. This newer version of the verifier will be more secure and will involve generating larger random numbers or using more complex hashing functions which can explain why you see issues while establishing JDBC connections. As mentioned by other responses using /dev/urandom normally resolves these issues. You can also decide to downgrade your password verifier and make the newer driver use the same older password verifier that your previous driver was using. For example if you want to use the 10G password verifier (for testing purposes only), first you need to make sure it's available for your user. Set SQLNET.ALLOWED_LOGON_VERSION_SERVER=8 in sqlnet.ora on the server. Then:

SQL> alter user scott identified by "tiger";

User altered.

SQL> SELECT PASSWORD_VERSIONS FROM DBA_USERS WHERE USERNAME='SCOTT';
PASSWORD_VERSIONS
-----------------
10G 11G 12C

Then you can force the JDBC thin driver to use the 10G verifier by setting this JDBC property oracle.jdbc.thinLogonCapability="o3". If you run into the error "ORA-28040: No matching authentication protocol" then that means your server is not allowing the 10G verifier to be used. If that's the case then you need to check your configuration again.

There is a solution provided to this problem in some of the OTN forums (https://kr.forums.oracle.com/forums/thread.jspa?messageID=3699989). But, the root cause of the problem is not explained. Following is my attempt to explain the root cause of the problem.

The Oracle JDBC drivers communicate with the Oracle server in a secure way. The drivers use the java.security.SecureRandom class to gather entropy for securing the communication. This class relies on the native platform support for gathering the entropy.

Entropy is the randomness collected/generated by an operating system or application for use in cryptography or other uses that require random data. This randomness is often collected from hardware sources, either from the hardware noises, audio data, mouse movements or specially provided randomness generators. The kernel gathers the entropy and stores it is an entropy pool and makes the random character data available to the operating system processes or applications through the special files /dev/random and /dev/urandom.

Reading from /dev/random drains the entropy pool with requested amount of bits/bytes, providing a high degree of randomness often desired in cryptographic operations. In case, if the entropy pool is completely drained and sufficient entropy is not available, the read operation on /dev/random blocks until additional entropy is gathered. Due to this, applications reading from /dev/random may block for some random period of time.

In contrast to the above, reading from the /dev/urandom does not block. Reading from /dev/urandom, too, drains the entropy pool but when short of sufficient entropy, it does not block but reuses the bits from the partially read random data. This is said to be susceptible to cryptanalytical attacks. This is a theorotical possibility and hence it is discouraged to read from /dev/urandom to gather randomness in cryptographic operations.

The java.security.SecureRandom class, by default, reads from the /dev/random file and hence sometimes blocks for random period of time. Now, if the read operation does not return for a required amount of time, the Oracle server times out the client (the jdbc drivers, in this case) and drops the communication by closing the socket from its end. The client when tries to resume the communication after returning from the blocking call encounters the IO exception. This problem may occur randomly on any platform, especially, where the entropy is gathered from hardware noises.

As suggested in the OTN forum, the solution to this problem is to override the default behaviour of java.security.SecureRandom class to use the non-blocking read from /dev/urandom instead of the blocking read from /dev/random. This can be done by adding the following system property -Djava.security.egd=file:///dev/urandom to the JVM. Though this is a good solution for the applications like the JDBC drivers, it is discouraged for applications that perform core cryptographic operations like crytographic key generation.

Other solutions could be to use different random seeder implementations available for the platform that do not rely on hardware noises for gathering entropy. With this, you may still require to override the default behaviour of java.security.SecureRandom.

Increasing the socket timeout on the Oracle server side can also be a solution but the side effects should be assessed from the server point of view before attempting this.

As per Bug https://bugs.openjdk.java.net/browse/JDK-6202721

Java will not consder -Djava.security.egd=file:/dev/urandom

It should be -Djava.security.egd=file:/dev/./urandom

A "connection reset" error message generally means that the other side has aborted the connection during the attempt to create a connection (the handshake). This has a lot of possible causes. A bug in the JDBC driver, a timeout at the DB side, a restart of the database, the DB being run out of available connections, poor network quality, bad virusscanner/firewall/proxy, etc.

As it happens intermittely, a bug in the JDBC driver can be less or more excluded. Left behind the remaining possible causes. I suggest to start with looking in the logs of the DB server.

Other thing that was causing me this problem was having the HOSTNAME settings wrong. My connection attempt was hanged at:

"main" prio=10 tid=0x00007f7cc8009000 nid=0x2f3a runnable [0x00007f7cce69e000]
   java.lang.Thread.State: RUNNABLE
        at java.net.Inet4AddressImpl.getLocalHostName(Native Method)
        at java.net.InetAddress.getLocalHost(InetAddress.java:1444)
        at sun.security.provider.SeedGenerator$1.run(SeedGenerator.java:176)
        at sun.security.provider.SeedGenerator$1.run(SeedGenerator.java:162)
        at java.security.AccessController.doPrivileged(Native Method)

So make sure you have an entry for your hostname in /etc/hosts/.

If you issue a hostname command like this:

$ hostname
my.server.com

You need a line in your /etc/hosts:

127.0.0.1 my my.server.com

Note that the suggested solution of using /dev/urandom did work the first time for me but didn't work always after that.

DBA at my firm switched of 'SQL* net banners' and that fixed it permanently for me with or without the above.

I don't know what 'SQL* net banners' are, but am hoping by putting this information here that if you have(are) a DBA he(you) would know what to do.

I was facing exactly the same problem. With Windows Vista I could not reproduce the problem but on Ubuntu I reproduced the 'connection reset'-Error constantly.

I found http://forums.oracle.com/forums/thread.jspa?threadID=941911&tstart=0&messageID=3793101

According to a user on that forum:

I opened a ticket with Oracle and this is what they told me.

java.security.SecureRandom is a standard API provided by sun. Among various methods offered by this class void nextBytes(byte[]) is one. This method is used for generating random bytes. Oracle 11g JDBC drivers use this API to generate random number during login. Users using Linux have been encountering SQLException("Io exception: Connection reset").

The problem is two fold

1. The JVM tries to list all the files in the /tmp (or alternate tmp directory set by -Djava.io.tmpdir) when SecureRandom.nextBytes(byte[]) is invoked. If the number of files is large the method takes a long time to respond and hence cause the server to timeout

2. The method void nextBytes(byte[]) uses /dev/random on Linux and on some machines which lack the random number generating hardware the operation slows down to the extent of bringing the whole login process to a halt. Ultimately the the user encounters SQLException("Io exception: Connection reset")

Users upgrading to 11g can encounter this issue if the underlying OS is Linux which is running on a faulty hardware.

Cause The cause of this has not yet been determined exactly. It could either be a problem in your hardware or the fact that for some reason the software cannot read from dev/random

Solution Change the setup for your application, so you add the next parameter to the java command:

-Djava.security.egd=file:/dev/../dev/urandom

We made this change in our java.security file and it has gotten rid of the error.

which solved my problem.