How can you search Google Programmatically Java API

Question

Does anyone know if and how it is possible to search Google programmatically - especially if there is a Java API for it

User · Answer

As an alternative to BalusC answer as it has been deprecated and you have to use proxies, you can use this package. Code sample:

Map<String, String> parameter = new HashMap<>();
parameter.put("q", "Coffee");
parameter.put("location", "Portland");
GoogleSearchResults serp = new GoogleSearchResults(parameter);

JsonObject data = serp.getJson();
JsonArray results = (JsonArray) data.get("organic_results");
JsonObject first_result = results.get(0).getAsJsonObject();
System.out.println("first coffee: " + first_result.get("title").getAsString());

Library on GitHub

User · Answer

In the Terms of Service of google we can read   5 3 You agree not to access  or attempt to access  any of the Services by any means other than through the interface that is provided by Google  unless you have been specifically allowed to do so in a separate agreement with Google  You specifically agree not to access  or attempt to access  any of the Services through any automated means  including use of scripts or web crawlers  and shall ensure that you comply with the instructions set out in any robots txt file present on the Services   So I guess the answer is No  More over the SOAP API is no longer available

User · Answer

Just an alternative  Searching google and parsing the results can also be done in a generic way using any HTML Parser such as Jsoup in Java  Following is the link to the mentioned example  Update  Link no longer works  Please look for any other example  https   www codeforeach com java example-how-to-search-google-using-java

User · Answer

Indeed there is an API to search google programmatically  The API is called google custom search  For using this API  you will need an Google Developer API key and a cx key  A simple procedure for accessing google search from java program is explained in my blog   Now dead  here is the Wayback Machine link

User · Answer

In light of those TOS alterations last year we built an API that gives access to Google s search  It was for our own use only but after some requests we decided to open it up  We re planning to add additional search engines in the future   Should anyone be looking for an easy way to implement   acquire search results you are free to sign up and give the REST API a try  https   searchapi io  It returns JSON results and should be easy enough to implement with the detailed docs   It s a shame that Bing and Yahoo are miles ahead on Google in this regard  Their APIs aren t cheap  but at least available

User · Answer

Some facts   Google offers a public search webservice API which returns JSON  http   ajax googleapis com ajax services search web  Documentation here  Java offers java net URL and java net URLConnection to fire and handle HTTP requests   JSON can in Java be converted to a fullworthy Javabean object using an arbitrary Java JSON API  One of the best is Google Gson    Now do the math  public static void main String   args  throws Exception       String google    quot http   ajax googleapis com ajax services search web v 1 0 amp q  quot       String search    quot stackoverflow quot       String charset    quot UTF-8 quot            URL url   new URL google   URLEncoder encode search  charset        Reader reader   new InputStreamReader url openStream    charset       GoogleResults results   new Gson   fromJson reader  GoogleResults class               Show title and URL of 1st result      System out println results getResponseData   getResults   get 0  getTitle         System out println results getResponseData   getResults   get 0  getUrl        With this Javabean class representing the most important JSON data as returned by Google  it actually returns more data  but it s left up to you as an exercise to expand this Javabean code accordingly   public class GoogleResults        private ResponseData responseData      public ResponseData getResponseData     return responseData        public void setResponseData ResponseData responseData    this responseData   responseData        public String toString     return  quot ResponseData  quot    responseData    quot   quot          static class ResponseData           private List lt Result gt  results          public List lt Result gt  getResults     return results            public void setResults List lt Result gt  results    this results   results            public String toString     return  quot Results  quot    results    quot   quot                static class Result           private String url          private String title          public String getUrl     return url            public String getTitle     return title            public void setUrl String url    this url   url            public void setTitle String title    this title   title            public String toString     return  quot Result url  quot    url   quot  title  quot    title    quot   quot                  See also   How to fire and handle HTTP requests using java net URLConnection How to convert JSON to Java   Update since November 2010  2 months after the above answer   the public search webservice has become deprecated  and the last day on which the service was offered was September 29  2014   Your best bet is now querying http   www google com search directly along with a honest user agent and then parse the result using a HTML parser  If you omit the user agent  then you get a 403 back  If you re lying in the user agent and simulate a web browser  e g  Chrome or Firefox   then you get a way much larger HTML response back which is a waste of bandwidth and performance  Here s a kickoff example using Jsoup as HTML parser  String google    quot http   www google com search q  quot   String search    quot stackoverflow quot   String charset    quot UTF-8 quot   String userAgent    quot ExampleBot 1 0   http   example com bot  quot      Change this to your company s name and bot homepage   Elements links   Jsoup connect google   URLEncoder encode search  charset   userAgent userAgent  get   select  quot  g gt  r gt a quot     for  Element link   links        String title   link text        String url   link absUrl  quot href quot       Google returns URLs in format  quot http   www google com url q  lt url gt  amp sa U amp ei  lt someKey gt  quot       url   URLDecoder decode url substring url indexOf        1  url indexOf   amp       quot UTF-8 quot             if   url startsWith  quot http quot              continue     Ads news etc                 System out println  quot Title   quot    title       System out println  quot URL   quot    url

User · Answer

To search google using API you should use Google Custom Search  scraping web page is not allowed  In java you can use CustomSearch API Client Library for Java  The maven dependency is    lt dependency gt       lt groupId gt com google apis lt  groupId gt       lt artifactId gt google-api-services-customsearch lt  artifactId gt       lt version gt v1-rev57-1 23 0 lt  version gt   lt  dependency gt     Example code searching using Google CustomSearch API Client Library  public static void main String   args  throws GeneralSecurityException  IOException        String searchQuery    test     The query to search     String cx    002845322276752338984 vxqzfa86nqc     Your search engine        Instance Customsearch     Customsearch cs   new Customsearch Builder GoogleNetHttpTransport newTrustedTransport    JacksonFactory getDefaultInstance    null                       setApplicationName  MyApplication                        setGoogleClientRequestInitializer new CustomsearchRequestInitializer  your api key                         build           Set search parameter     Customsearch Cse List list   cs cse   list searchQuery  setCx cx           Execute search     Search result   list execute        if  result getItems    null           for  Result ri   result getItems                    Get title  link  body etc  from search             System out println ri getTitle            ri getLink                          As you can see you will need to request an api key and setup an own search engine id  cx       Note that you can search the whole web by selecting  Search entire web  on basic tab settings during setup of cx  but results will not be exactly the same as a normal browser google search    Currently  date of answer  you get 100 api calls per day for free  then google like to share your profit

User · Answer

Google TOS have been relaxed a bit in April 2014  Now it states     Don   t misuse our Services  For example  don   t interfere with our Services or try to access them using a method other than the interface and the instructions that we provide     So the passage about  automated means  and scripts is gone now  It evidently still is not the desired  by google  way of accessing their services  but I think it is now formally open to interpretation of what exactly an  interface  is and whether it makes any difference as of how exactly returned HTML is processed  rendered or parsed   Anyhow  I have written a Java convenience library and it is up to you to decide whether to use it or not   https   github com afedulov google-web-search

[java] How can you search Google Programmatically Java API

Examples related to java

Examples related to google-search-api