Best Practice to Use HttpClient in Multithreaded Environment

Question

For a while  I have been using HttpClient in a multithreaded environment  For every thread  when it initiates a connection  it will create a completely new HttpClient instance   Recently  I have discovered that  by using this approach  it can cause the user to have too many ports being opened  and most of the connections are in TIME WAIT state   http   www opensubscriber com message commons-httpclient-dev jakarta apache org 86045 html  Hence  instead of each thread doing    HttpClient c   new HttpClient    try       c executeMethod method     catch          finally       method releaseConnection        We plan to have     METHOD A      global c is initialized once through    HttpClient global c   new HttpClient new MultiThreadedHttpConnectionManager      try       global c executeMethod method     catch          finally       method releaseConnection        In a normal situation  global c will be accessed by 50   threads concurrently  I was wondering  will this create any performance issues  Is MultiThreadedHttpConnectionManager using a lock-free mechanism to implement its thread safe policy    If 10 threads are using global c  will the other 40 threads be locked   Or would it be better if  in every thread  I create an instance of an HttpClient  but release the connection manager explicitly    METHOD B   MultiThreadedHttpConnectionManager connman   new MultiThreadedHttpConnectionManager    HttpClient c   new HttpClient connman   try         c executeMethod method     catch          finally       method releaseConnection        connman shutdown        Will connman shutdown   suffer performance issues   May I know which method  A or B  is better  for application using an 50   threads

User · Answer

My reading of the docs is that HttpConnection itself is not treated as thread safe, and hence MultiThreadedHttpConnectionManager provides a reusable pool of HttpConnections, you have a single MultiThreadedHttpConnectionManager shared by all threads and initialised exactly once. So you need a couple of small refinements to option A.

MultiThreadedHttpConnectionManager connman = new MultiThreadedHttpConnectionManag

Then each thread should be using the sequence for every request, getting a conection from the pool and putting it back on completion of its work - using a finally block may be good. You should also code for the possibility that the pool has no available connections and process the timeout exception.

HttpConnection connection = null
try {
    connection = connman.getConnectionWithTimeout(
                        HostConfiguration hostConfiguration, long timeout) 
    // work
} catch (/*etc*/) {/*etc*/} finally{
    if ( connection != null )
        connman.releaseConnection(connection);
}

As you are using a pool of connections you won't actually be closing the connections and so this should not hit the TIME_WAIT problem. This approach does assuume that each thread doesn't hang on to the connection for long. Note that conman itself is left open.

User · Answer

With HttpClient 4 5 you can do this   CloseableHttpClient httpClient   HttpClients custom   setConnectionManager new PoolingHttpClientConnectionManager    build      Note that this one implements Closeable  for shutting down of the connection manager

User · Answer

I think you will want to use ThreadSafeClientConnManager   You can see how it works here  http   foo jasonhudgins com 2009 08 http-connection-reuse-in-android html  Or in the AndroidHttpClient which uses it internally

User · Answer

Definitely Method A because its pooled and thread safe   If you are using httpclient 4 x  the connection manager is called ThreadSafeClientConnManager  See this link for further details  scroll down to  Pooling connection manager    For example       HttpParams params   new BasicHttpParams        SchemeRegistry registry   new SchemeRegistry        registry register new Scheme  http   PlainSocketFactory getSocketFactory    80        ClientConnectionManager cm   new ThreadSafeClientConnManager params  registry       HttpClient client   new DefaultHttpClient cm  params

User · Answer

Method A is recommended by httpclient developer community   Please refer http   www mail-archive com httpclient-users hc apache org msg02455 html for more details

[java] Best Practice to Use HttpClient in Multithreaded Environment

Examples related to java

Examples related to apache-commons-httpclient