How to use java net URLConnection to fire and handle HTTP requests

Question

Use of java net URLConnection is asked about pretty often here  and the Oracle tutorial is too concise about it    That tutorial basically only shows how to fire a GET request and read the response  It doesn t explain anywhere how to use it to among others perform a POST request  set request headers  read response headers  deal with cookies  submit a HTML form  upload a file  etc    So  how can I use java net URLConnection to fire and handle  advanced  HTTP requests

User · Answer

You can also use JdkRequest from jcabi-http  I m a developer   which does all this work for you  decorating HttpURLConnection  firing HTTP requests and parsing responses  for example   String html   new JdkRequest  http   www google com   fetch   body      Check this blog post for more info  http   www yegor256 com 2014 04 11 jcabi-http-intro html

User · Answer

Update     The new HTTP Client shipped with Java 9 but as part of an   Incubator module named jdk incubator httpclient  Incubator modules are   a means of putting non-final APIs in the hands of developers while the   APIs progress towards either finalization or removal in a future   release    In Java 9  you can send a GET request like      GET HttpResponse response   HttpRequest      create new URI  http   www stackoverflow com         headers  Foo    foovalue    Bar    barvalue        GET        response      Then you can examine the returned HttpResponse   int statusCode   response statusCode    String responseBody   response body HttpResponse asString       Since this new HTTP Client is in java httpclient jdk incubator httpclient module  you should declare this dependency in your module-info java file   module com foo bar       requires jdk incubator httpclient

User · Answer

Inspired by this and other questions on SO  I ve created a minimal open source basic-http-client that embodies most of the techniques found here   google-http-java-client is also a great open source resource

User · Answer

Initially I was misled by this article which favours HttpClient  Later I have been realized that HttpURLConnection is going to stay from this article As per the Google blog   Apache HTTP client has fewer bugs on Eclair and Froyo  It is the best choice for these releases  For Gingerbread   HttpURLConnection is the best choice  Its simple API and small size makes it great fit for Android  Transparent compression and response caching reduce network use  improve speed and save battery  New applications should use HttpURLConnection  it is where we will be spending our energy going forward   After reading this article and some other stack over flow questions  I am convinced that HttpURLConnection is going to stay for longer durations  Some of the SE questions favouring HttpURLConnections  On Android  make a POST request with URL Encoded Form data without using UrlEncodedFormEntity HttpPost works in Java project  not in Android

User · Answer

There is also OkHttp  which is an HTTP client that   s efficient by default         HTTP 2 support allows all requests to the same host to share a    socket    Connection pooling reduces request latency  if HTTP 2 isn   t    available     Transparent GZIP shrinks download sizes    Response caching avoids the network completely for repeat requests       First create an instance of OkHttpClient   OkHttpClient client   new OkHttpClient      Then  prepare your GET request   Request request   new Request Builder          url url         build      finally  use OkHttpClient to send prepared Request   Response response   client newCall request  execute      For more details  you can consult the OkHttp s documentation

User · Answer

I was also very inspired by this response   I am often on projects where I need to do some HTTP  and I may not want to bring in a lot of 3rd party dependencies  which bring in others and so on and so on  etc    I started to write my own utilities based on some of this conversation  not any where done    package org boon utils    import java io IOException  import java io InputStream  import java net HttpURLConnection  import java net URL  import java net URLConnection  import java util Map   import static org boon utils IO read   public class HTTP     Then there are just a bunch or static methods   public static String get          final String url         Exceptions tryIt    - gt            URLConnection connection          connection   doGet url  null  null  null           return extractResponseString connection               return null     public static String getWithHeaders          final String url          final Map lt String    extends Object gt  headers        URLConnection connection      try           connection   doGet url  headers  null  null           return extractResponseString connection         catch  Exception ex            Exceptions handle ex           return null           public static String getWithContentType          final String url          final Map lt String    extends Object gt  headers          String contentType        URLConnection connection      try           connection   doGet url  headers  contentType  null           return extractResponseString connection         catch  Exception ex            Exceptions handle ex           return null          public static String getWithCharSet          final String url          final Map lt String    extends Object gt  headers          String contentType          String charSet        URLConnection connection      try           connection   doGet url  headers  contentType  charSet           return extractResponseString connection         catch  Exception ex            Exceptions handle ex           return null            Then post     public static String postBody          final String url          final String body        URLConnection connection      try           connection   doPost url  null   text plain   null  body           return extractResponseString connection         catch  Exception ex            Exceptions handle ex           return null           public static String postBodyWithHeaders          final String url          final Map lt String    extends Object gt  headers          final String body        URLConnection connection      try           connection   doPost url  headers   text plain   null  body           return extractResponseString connection         catch  Exception ex            Exceptions handle ex           return null             public static String postBodyWithContentType          final String url          final Map lt String    extends Object gt  headers          final String contentType          final String body         URLConnection connection      try           connection   doPost url  headers  contentType  null  body             return extractResponseString connection           catch  Exception ex            Exceptions handle ex           return null              public static String postBodyWithCharset          final String url          final Map lt String    extends Object gt  headers          final String contentType          final String charSet          final String body         URLConnection connection      try           connection   doPost url  headers  contentType  charSet  body             return extractResponseString connection           catch  Exception ex            Exceptions handle ex           return null             private static URLConnection doPost String url  Map lt String    extends Object gt  headers                                      String contentType  String charset  String body                                       throws IOException       URLConnection connection    Handle output         connection   new URL url  openConnection        connection setDoOutput true       manageContentTypeHeaders contentType  charset  connection        manageHeaders headers  connection         IO write connection getOutputStream    body  IO CHARSET       return connection     private static void manageHeaders Map lt String    extends Object gt  headers  URLConnection connection        if  headers    null            for  Map Entry lt String    extends Object gt  entry   headers entrySet                  connection setRequestProperty entry getKey    entry getValue   toString                        private static void manageContentTypeHeaders String contentType  String charset  URLConnection connection        connection setRequestProperty  Accept-Charset   charset    null   IO CHARSET   charset       if  contentType  null  amp  amp   contentType isEmpty              connection setRequestProperty  Content-Type   contentType            private static URLConnection doGet String url  Map lt String    extends Object gt  headers                                      String contentType  String charset  throws IOException       URLConnection connection    Handle output         connection   new URL url  openConnection        manageContentTypeHeaders contentType  charset  connection        manageHeaders headers  connection        return connection     private static String extractResponseString URLConnection connection  throws IOException      Handle input         HttpURLConnection http    HttpURLConnection connection      int status   http getResponseCode        String charset   getCharset connection getHeaderField  Content-Type          if  status  200            return readResponseBody http  charset         else           return readErrorResponseBody http  status  charset            private static String readErrorResponseBody HttpURLConnection http  int status  String charset        InputStream errorStream   http getErrorStream        if   errorStream  null             String error   charset   null   read  errorStream                 read  errorStream  charset            throw new RuntimeException  STATUS CODE      status     n n    error         else           throw new RuntimeException  STATUS CODE      status            private static String readResponseBody HttpURLConnection http  String charset  throws IOException       if  charset    null            return read http getInputStream    charset         else           return read http getInputStream              private static String getCharset String contentType        if  contentType  null             return null            String charset   null      for  String param   contentType replace          split                 if  param startsWith  charset                   charset   param split      2  1               break                      charset   charset    null    IO CHARSET   charset       return charset      Well you get the idea      Here are the tests   static class MyHandler implements HttpHandler       public void handle HttpExchange t  throws IOException            InputStream requestBody   t getRequestBody            String body   IO read requestBody           Headers requestHeaders   t getRequestHeaders            body   body     n    copy requestHeaders  toString            t sendResponseHeaders 200  body length             OutputStream os   t getResponseBody            os write body getBytes             os close               Test public void testHappy   throws Exception        HttpServer server   HttpServer create new InetSocketAddress 9212   0       server createContext   test   new MyHandler         server setExecutor null      creates a default executor     server start         Thread sleep 10         Map lt String String gt  headers   map  foo    bar    fun    sun         String response   HTTP postBodyWithContentType  http   localhost 9212 test   headers   text plain    hi mom         System out println response        assertTrue response contains  hi mom         assertTrue response contains  Fun  sun   Foo  bar            response   HTTP postBodyWithCharset  http   localhost 9212 test   headers   text plain    UTF-8    hi mom         System out println response        assertTrue response contains  hi mom         assertTrue response contains  Fun  sun   Foo  bar           response   HTTP postBodyWithHeaders  http   localhost 9212 test   headers   hi mom         System out println response        assertTrue response contains  hi mom         assertTrue response contains  Fun  sun   Foo  bar            response   HTTP get  http   localhost 9212 test         System out println response         response   HTTP getWithHeaders  http   localhost 9212 test   headers        System out println response        assertTrue response contains  Fun  sun   Foo  bar             response   HTTP getWithContentType  http   localhost 9212 test   headers   text plain         System out println response        assertTrue response contains  Fun  sun   Foo  bar             response   HTTP getWithCharSet  http   localhost 9212 test   headers   text plain    UTF-8         System out println response        assertTrue response contains  Fun  sun   Foo  bar           Thread sleep 10        server stop 0         Test public void testPostBody   throws Exception        HttpServer server   HttpServer create new InetSocketAddress 9220   0       server createContext   test   new MyHandler         server setExecutor null      creates a default executor     server start         Thread sleep 10         Map lt String String gt  headers   map  foo    bar    fun    sun         String response   HTTP postBody  http   localhost 9220 test    hi mom         assertTrue response contains  hi mom           Thread sleep 10        server stop 0         Test expected   RuntimeException class  public void testSad   throws Exception        HttpServer server   HttpServer create new InetSocketAddress 9213   0       server createContext   test   new MyHandler         server setExecutor null      creates a default executor     server start         Thread sleep 10         Map lt String String gt  headers   map  foo    bar    fun    sun         String response   HTTP postBodyWithContentType  http   localhost 9213 foo   headers   text plain    hi mom         System out println response        assertTrue response contains  hi mom         assertTrue response contains  Fun  sun   Foo  bar           Thread sleep 10        server stop 0         You can find the rest here   https   github com RichardHightower boon  My goal is to provide the common things one would want to do in a bit more easier way then

User · Answer

First a disclaimer beforehand  the posted code snippets are all basic examples  You ll need to handle trivial IOExceptions and RuntimeExceptions like NullPointerException  ArrayIndexOutOfBoundsException and consorts yourself   Preparing We first need to know at least the URL and the charset  The parameters are optional and depend on the functional requirements  String url    quot http   example com quot   String charset    quot UTF-8 quot       Or in Java 7 and later  use the constant  java nio charset StandardCharsets UTF 8 name   String param1    quot value1 quot   String param2    quot value2 quot           String query   String format  quot param1  s amp param2  s quot         URLEncoder encode param1  charset         URLEncoder encode param2  charset     The query parameters must be in name value format and be concatenated by  amp   You would normally also URL-encode the query parameters with the specified charset using URLEncoder encode    The String format   is just for convenience  I prefer it when I would need the String concatenation operator   more than twice   Firing an HTTP GET request with  optionally  query parameters It s a trivial task  It s the default request method  URLConnection connection   new URL url    quot   quot    query  openConnection    connection setRequestProperty  quot Accept-Charset quot   charset   InputStream response   connection getInputStream            Any query string should be concatenated to the URL using    The Accept-Charset header may hint the server what encoding the parameters are in  If you don t send any query string  then you can leave the Accept-Charset header away  If you don t need to set any headers  then you can even use the URL openStream   shortcut method  InputStream response   new URL url  openStream            Either way  if the other side is an HttpServlet  then its doGet   method will be called and the parameters will be available by HttpServletRequest getParameter    For testing purposes  you can print the response body to stdout as below  try  Scanner scanner   new Scanner response         String responseBody   scanner useDelimiter  quot   A quot   next        System out println responseBody       Firing an HTTP POST request with query parameters Setting the URLConnection setDoOutput   to true implicitly sets the request method to POST  The standard HTTP POST as web forms do is of type application x-www-form-urlencoded wherein the query string is written to the request body  URLConnection connection   new URL url  openConnection    connection setDoOutput true      Triggers POST  connection setRequestProperty  quot Accept-Charset quot   charset   connection setRequestProperty  quot Content-Type quot    quot application x-www-form-urlencoded charset  quot    charset    try  OutputStream output   connection getOutputStream          output write query getBytes charset       InputStream response   connection getInputStream            Note  whenever you d like to submit a HTML form programmatically  don t forget to take the name value pairs of any  lt input type  quot hidden quot  gt  elements into the query string and of course also the name value pair of the  lt input type  quot submit quot  gt  element which you d like to  quot press quot  programmatically  because that s usually been used in the server side to distinguish if a button was pressed and if so  which one   You can also cast the obtained URLConnection to HttpURLConnection and use its HttpURLConnection setRequestMethod   instead  But if you re trying to use the connection for output you still need to set URLConnection setDoOutput   to true  HttpURLConnection httpConnection    HttpURLConnection  new URL url  openConnection    httpConnection setRequestMethod  quot POST quot            Either way  if the other side is an HttpServlet  then its doPost   method will be called and the parameters will be available by HttpServletRequest getParameter     Actually firing the HTTP request You can fire the HTTP request explicitly with URLConnection connect    but the request will automatically be fired on demand when you want to get any information about the HTTP response  such as the response body using URLConnection getInputStream   and so on  The above examples does exactly that  so the connect   call is in fact superfluous   Gathering HTTP response information  HTTP response status   You need an HttpURLConnection here  Cast it first if necessary      int status   httpConnection getResponseCode      HTTP response headers   for  Entry lt String  List lt String gt  gt  header   connection getHeaderFields   entrySet           System out println header getKey      quot   quot    header getValue          HTTP response encoding    When the Content-Type contains a charset parameter  then the response body is likely text based and we d like to process the response body with the server-side specified character encoding then      String contentType   connection getHeaderField  quot Content-Type quot        String charset   null       for  String param   contentType replace  quot   quot    quot  quot   split  quot   quot              if  param startsWith  quot charset  quot                  charset   param split  quot   quot   2  1               break                       if  charset    null            try  BufferedReader reader   new BufferedReader new InputStreamReader response  charset                  for  String line   line   reader readLine       null                            System out println line                                  else              It s likely binary content  use InputStream OutputStream          Maintaining the session The server side session is usually backed by a cookie  Some web forms require that you re logged in and or are tracked by a session  You can use the CookieHandler API to maintain cookies  You need to prepare a CookieManager with a CookiePolicy of ACCEPT ALL before sending all HTTP requests     First set the default cookie manager  CookieHandler setDefault new CookieManager null  CookiePolicy ACCEPT ALL        All the following subsequent URLConnections will use the same cookie manager  URLConnection connection   new URL url  openConnection            connection   new URL url  openConnection            connection   new URL url  openConnection            Note that this is known to not always work properly in all circumstances  If it fails for you  then best is to manually gather and set the cookie headers  You basically need to grab all Set-Cookie headers from the response of the login or the first GET request and then pass this through the subsequent requests     Gather all cookies on the first request  URLConnection connection   new URL url  openConnection    List lt String gt  cookies   connection getHeaderFields   get  quot Set-Cookie quot               Then use the same cookies on all subsequent requests  connection   new URL url  openConnection    for  String cookie   cookies        connection addRequestProperty  quot Cookie quot   cookie split  quot   quot   2  0              The split  quot   quot   2  0  is there to get rid of cookie attributes which are irrelevant for the server side like expires  path  etc  Alternatively  you could also use cookie substring 0  cookie indexOf       instead of split     Streaming mode The HttpURLConnection will by default buffer the entire request body before actually sending it  regardless of whether you ve set a fixed content length yourself using connection setRequestProperty  quot Content-Length quot   contentLength    This may cause OutOfMemoryExceptions whenever you concurrently send large POST requests  e g  uploading files   To avoid this  you would like to set the HttpURLConnection setFixedLengthStreamingMode    httpConnection setFixedLengthStreamingMode contentLength    But if the content length is really not known beforehand  then you can make use of chunked streaming mode by setting the HttpURLConnection setChunkedStreamingMode   accordingly  This will set the HTTP Transfer-Encoding header to chunked which will force the request body being sent in chunks  The below example will send the body in chunks of 1KB  httpConnection setChunkedStreamingMode 1024     User-Agent It can happen that a request returns an unexpected response  while it works fine with a real web browser  The server side is probably blocking requests based on the User-Agent request header  The URLConnection will by default set it to Java 1 6 0 19 where the last part is obviously the JRE version  You can override this as follows  connection setRequestProperty  quot User-Agent quot    quot Mozilla 5 0  Windows NT 6 1  AppleWebKit 537 36  KHTML  like Gecko  Chrome 41 0 2228 0 Safari 537 36 quot       Do as if you re using Chrome 41 on Windows 7   Use the User-Agent string from a recent browser   Error handling If the HTTP response code is 4nn  Client Error  or 5nn  Server Error   then you may want to read the HttpURLConnection getErrorStream   to see if the server has sent any useful error information  InputStream error     HttpURLConnection  connection  getErrorStream     If the HTTP response code is -1  then something went wrong with connection and response handling  The HttpURLConnection implementation is in older JREs somewhat buggy with keeping connections alive  You may want to turn it off by setting the http keepAlive system property to false  You can do this programmatically in the beginning of your application by  System setProperty  quot http keepAlive quot    quot false quot      Uploading files You d normally use multipart form-data encoding for mixed POST content  binary and character data   The encoding is in more detail described in RFC2388  String param    quot value quot   File textFile   new File  quot  path to file txt quot    File binaryFile   new File  quot  path to file bin quot    String boundary   Long toHexString System currentTimeMillis        Just generate some unique random value  String CRLF    quot  r n quot      Line separator required by multipart form-data  URLConnection connection   new URL url  openConnection    connection setDoOutput true   connection setRequestProperty  quot Content-Type quot    quot multipart form-data  boundary  quot    boundary    try       OutputStream output   connection getOutputStream        PrintWriter writer   new PrintWriter new OutputStreamWriter output  charset   true              Send normal param      writer append  quot -- quot    boundary  append CRLF       writer append  quot Content-Disposition  form-data  name   quot param  quot  quot   append CRLF       writer append  quot Content-Type  text plain  charset  quot    charset  append CRLF       writer append CRLF  append param  append CRLF  flush            Send text file      writer append  quot -- quot    boundary  append CRLF       writer append  quot Content-Disposition  form-data  name   quot textFile  quot   filename   quot  quot    textFile getName      quot   quot  quot   append CRLF       writer append  quot Content-Type  text plain  charset  quot    charset  append CRLF      Text file itself must be saved in this charset      writer append CRLF  flush        Files copy textFile toPath    output       output flush       Important before continuing with writer      writer append CRLF  flush       CRLF is important  It indicates end of boundary          Send binary file      writer append  quot -- quot    boundary  append CRLF       writer append  quot Content-Disposition  form-data  name   quot binaryFile  quot   filename   quot  quot    binaryFile getName      quot   quot  quot   append CRLF       writer append  quot Content-Type   quot    URLConnection guessContentTypeFromName binaryFile getName     append CRLF       writer append  quot Content-Transfer-Encoding  binary quot   append CRLF       writer append CRLF  flush        Files copy binaryFile toPath    output       output flush       Important before continuing with writer      writer append CRLF  flush       CRLF is important  It indicates end of boundary          End of multipart form-data      writer append  quot -- quot    boundary    quot -- quot   append CRLF  flush       If the other side is an HttpServlet  then its doPost   method will be called and the parts will be available by HttpServletRequest getPart    note  thus not getParameter   and so on    The getPart   method is however relatively new  it s introduced in Servlet 3 0  Glassfish 3  Tomcat 7  etc   Prior to Servlet 3 0  your best choice is using Apache Commons FileUpload to parse a multipart form-data request  Also see this answer for examples of both the FileUpload and the Servelt 3 0 approaches   Dealing with untrusted or misconfigured HTTPS sites Sometimes you need to connect an HTTPS URL  perhaps because you re writing a web scraper  In that case  you may likely face a javax net ssl SSLException  Not trusted server certificate on some HTTPS sites who doesn t keep their SSL certificates up to date  or a java security cert CertificateException  No subject alternative DNS name matching  hostname  found or javax net ssl SSLProtocolException  handshake alert  unrecognized name on some misconfigured HTTPS sites  The following one-time-run static initializer in your web scraper class should make HttpsURLConnection more lenient as to those HTTPS sites and thus not throw those exceptions anymore  static       TrustManager   trustAllCertificates   new TrustManager             new X509TrustManager                  Override             public X509Certificate   getAcceptedIssuers                     return null     Not relevant                             Override             public void checkClientTrusted X509Certificate   certs  String authType                       Do nothing  Just allow them all                             Override             public void checkServerTrusted X509Certificate   certs  String authType                       Do nothing  Just allow them all                                      HostnameVerifier trustAllHostnames   new HostnameVerifier              Override         public boolean verify String hostname  SSLSession session                return true     Just allow them all                        try           System setProperty  quot jsse enableSNIExtension quot    quot false quot            SSLContext sc   SSLContext getInstance  quot SSL quot            sc init null  trustAllCertificates  new SecureRandom             HttpsURLConnection setDefaultSSLSocketFactory sc getSocketFactory             HttpsURLConnection setDefaultHostnameVerifier trustAllHostnames             catch  GeneralSecurityException e            throw new ExceptionInInitializerError e             Last words The Apache HttpComponents HttpClient is much more convenient in this all     HttpClient Tutorial HttpClient Examples   Parsing and extracting HTML If all you want is parsing and extracting data from HTML  then better use a HTML parser like Jsoup  What are the pros cons of leading HTML parsers in Java How to scan and extract a webpage in Java

User · Answer

When working with HTTP it s almost always more useful to refer to HttpURLConnection rather than the base class URLConnection  since URLConnection is an abstract class when you ask for URLConnection openConnection   on a HTTP URL that s what you ll get back anyway    Then you can instead of relying on URLConnection setDoOutput true  to implicitly set the request method to POST instead do httpURLConnection setRequestMethod  POST   which some might find more natural  and which also allows you to specify other request methods such as PUT  DELETE         It also provides useful HTTP constants so you can do   int responseCode   httpURLConnection getResponseCode     if  responseCode    HttpURLConnection HTTP OK

User · Answer

There are 2 options you can go with HTTP URL Hits   GET   POST  GET Request  -  HttpURLConnection setFollowRedirects true      defaults to true  String url    https   name of the url   URL request url   new URL url   HttpURLConnection http conn    HttpURLConnection request url openConnection    http conn setConnectTimeout 100000   http conn setReadTimeout 100000   http conn setInstanceFollowRedirects true   System out println String valueOf http conn getResponseCode        POST request  -  HttpURLConnection setFollowRedirects true      defaults to true  String url    https   name of the url  URL request url   new URL url   HttpURLConnection http conn    HttpURLConnection request url openConnection    http conn setConnectTimeout 100000   http conn setReadTimeout 100000   http conn setInstanceFollowRedirects true   http conn setDoOutput true   PrintWriter out   new PrintWriter http conn getOutputStream     if  urlparameter    null       out println urlparameter     out close    out   null  System out println String valueOf http conn getResponseCode

User · Answer

I suggest you take a look at the code on kevinsawicki http-request  its basically a wrapper on top of HttpUrlConnection it provides a much simpler API in case you just want to make the requests right now or you can take a look at the sources  it s not too big  to take a look at how connections are handled   Example  Make a GET request with content type application json and some query parameters      GET http   google com q baseball 20gloves amp size 100 String response   HttpRequest get  http   google com   true   q    baseball gloves    size   100           accept  application json            body    System out println  Response was      response

User · Answer

if you are using http get please remove this line  urlConnection setDoOutput true

[java] How to use java.net.URLConnection to fire and handle HTTP requests?

Examples related to java

Examples related to http

Examples related to httprequest

Examples related to httpurlconnection

Examples related to urlconnection