JsonParseException Unrecognized token http was expecting true false or null

Question

We have the following string which is a valid JSON written to a file on HDFS           id   tag search twitter com 2005 564407444843950080      objectType   activity      actor           objectType   person        id   id twitter com 2302910022        link   http 3A 2F 2Fwww twitter com 2Fme7me4610012        displayName           postedTime   2014-01-21T11 06 06 000Z        image   https 3A 2F 2Fpbs twimg com 2Fprofile images 2F563125491159162881 2FfypkHK3M normal jpeg        summary                                                                                                      lloooo 20        links                         href  null           rel   me                      friendsCount  10503       followersCount  10325       listedCount  12       statusesCount  84957       twitterTimeZone  null       verified  false       utcOffset  null       preferredUsername   me7me4610012        languages             ar              favoritesCount  17         verb   share      postedTime   2015-02-08T12 56 35 000Z      generator           displayName   Twitter for Android        link   http 3A 2F 2Ftwitter com 2Fdownload 2Fandroid          provider           objectType   service        displayName   Twitter        link   http 3A 2F 2Fwww twitter com          link   http 3A 2F 2Ftwitter com 2Fme7me4610012 2Fstatuses 2F564407444843950080      body   RT  sckud1                                                                                       http 3A 2F 2Ft co 2FC55SaQKmUV http 3A 2F 2Ft co 2Ft5TjIln         object           id   tag search twitter com 2005 564407126526013440        objectType   activity        actor             objectType   person          id   id twitter com 462268717          link   http 3A 2F 2Fwww twitter com sckud1          displayName                      postedTime   2012-01-12T19 24 17 000Z          image   https 3A 2F 2Fpbs twimg com 2Fprofile images 2F508424482885615616 2FmPBGZBPx normal jpeg          summary                                                                      http 3A 2F 2Fmarketgulf com          links                             href   http 3A 2F 2Fmarketgulf com              rel   me                            friendsCount  435237         followersCount  464951         listedCount  708         statusesCount  1071685         twitterTimeZone   Riyadh          verified  false         utcOffset   10800          preferredUsername   sckud1          languages               ar                  location               objectType   place            displayName   Made in K S A                  favoritesCount  77             verb   post        postedTime   2015-02-08T12 55 19 000Z        generator             displayName   Tweet Old Post          link   http 3A 2F 2Fwww ajaymatharu com 2F              provider             objectType   service          displayName   Twitter          link   http 3A 2F 2Fwww twitter com              link   http 3A 2F 2Ftwitter com 2Fsckud1 2Fstatuses 2F564407126526013440        body                                                                                        http 3A 2F 2Ft co 2FC55SaQKmUV http 3A 2F 2Ft co 2Ft5TjIlnZgN        object             objectType   note          id   object search twitter com 2005 564407126526013440          summary                                                                                        http 3A 2F 2Ft co 2FC55SaQKmUV http 3A 2F 2Ft co 2Ft5TjIlnZgN          link   http 3A 2F 2Ftwitter com 2Fsckud1 2Fstatuses 2F564407126526013440          postedTime   2015-02-08T12 55 19 000Z              favoritesCount  0       twitter entities             hashtags                       trends                       urls                             url   http 3A 2F 2Ft co 2FC55SaQKmUV              expanded url   http 3A 2F 2Fwww hasterya com 2Farchives 2F34688utm source 3DReviveOldPost 26utm medium 3Dsocial 26utm campaign 3DReviveOldPost              display url   hasterya com archives 34688                 indices                  85              107                                       user mentions                       symbols                       media                             id  564407126341468160             id str   564407126341468160              indices                  108              130                         media url   http 3A 2F 2Fpbs twimg com 2Fmedia 2FB9UtSoJIQAA07-r jpg              media url https   https 3A 2F 2Fpbs twimg com 2Fmedia 2FB9UtSoJIQAA07-r jpg              url   http 3A 2F 2Ft co 2Ft5TjIlnZgN              display url   pic twitter com t5TjIlnZgN              expanded url   http 3A 2F 2Ftwitter com 2Fsckud1 2Fstatus 2F564407126526013440 2Fphoto 2F1              type   photo              sizes                   large                     w  320                 h  180                 resize   fit                              thumb                     w  150                 h  150                 resize   crop                              small                     w  320                 h  180                 resize   fit                              medium                     w  320                 h  180                 resize   fit                                                          twitter extended entities             media                             id  564407126341468160             id str   564407126341468160              indices                  108              130                         media url   http 3A 2F 2Fpbs twimg com 2Fmedia 2FB9UtSoJIQAA07-r jpg              media url https   https 3A 2F 2Fpbs twimg com 2Fmedia 2FB9UtSoJIQAA07-r jpg              url   http 3A 2F 2Ft co 2Ft5TjIlnZgN              display url   pic twitter com t5TjIlnZgN              expanded url   http 3A 2F 2Ftwitter com 2Fsckud1 2Fstatus 2F564407126526013440 2Fphoto 2F1              type   photo              sizes                   large                     w  320                 h  180                 resize   fit                              thumb                     w  150                 h  150                 resize   crop                              small                     w  320                 h  180                 resize   fit                              medium                     w  320                 h  180                 resize   fit                                                          twitter filter level   low        twitter lang   ar          favoritesCount  0     twitter entities           hashtags                   trends                   urls                         url   http 3A 2F 2Ft co 2FC55SaQKmUV            expanded url   http 3A 2F 2Fwww hasterya com 2Farchives 2F34688utm source 3DReviveOldPost 26utm medium 3Dsocial 26utm campaign 3DReviveOldPost            display url   hasterya com archives 34688               indices                97            119                               user mentions                         screen name   sckud1            name                        id  462268717           id str   462268717            indices                3            10                               symbols                   media                         id  564407126341468160           id str   564407126341468160            indices                139            140                     media url   http 3A 2F 2Fpbs twimg com 2Fmedia 2FB9UtSoJIQAA07-r jpg            media url https   https 3A 2F 2Fpbs twimg com 2Fmedia 2FB9UtSoJIQAA07-r jpg            url   http 3A 2F 2Ft co 2Ft5TjIlnZgN            display url   pic twitter com t5TjIlnZgN            expanded url   http 3A 2F 2Ftwitter com 2Fsckud1 2Fstatus 2F564407126526013440 2Fphoto 2F1            type   photo            sizes                 large                   w  320               h  180               resize   fit                          thumb                   w  150               h  150               resize   crop                          small                   w  320               h  180               resize   fit                          medium                   w  320               h  180               resize   fit                                  source status id  564407126526013440           source status id str   564407126526013440                        twitter extended entities           media                         id  564407126341468160           id str   564407126341468160            indices                139            140                     media url   http 3A 2F 2Fpbs twimg com 2Fmedia 2FB9UtSoJIQAA07-r jpg            media url https   https 3A 2F 2Fpbs twimg com 2Fmedia 2FB9UtSoJIQAA07-r jpg            url   http 3A 2F 2Ft co 2Ft5TjIlnZgN            display url   pic twitter com t5TjIlnZgN            expanded url   http 3A 2F 2Ftwitter com 2Fsckud1 2Fstatus 2F564407126526013440 2Fphoto 2F1            type   photo            sizes                 large                   w  320               h  180               resize   fit                          thumb                   w  150               h  150               resize   crop                          small                   w  320               h  180               resize   fit                          medium                   w  320               h  180               resize   fit                                  source status id  564407126526013440           source status id str   564407126526013440                        twitter filter level   low      twitter lang   ar      retweetCount  1     gnip           matching rules                         tag   ISIS66                      urls                         url   http 3A 2F 2Ft co 2Ft5TjIlnZgN            expanded url   http 3A 2F 2Ftwitter com 2Fsckud1 2Fstatus 2F564407126526013440 2Fphoto 2F1            expanded status  200                             url   http 3A 2F 2Ft co 2FC55SaQKmUV            expanded url   http 3A 2F 2Fwww hasterya com 2Farchives 2F34688utm source 3DReviveOldPost 26utm medium 3Dsocial 26utm campaign 3DReviveOldPost            expanded status  200                     klout score  50       language             value   ar                EDIT  We configure a flume agent that reads the data from that file and pass it to Solr sink but unfortunately this exception in the title is throw   and here is the stack trace   org kitesdk morphline api MorphlineRuntimeException  com fasterxml jackson core JsonParseException  Unrecognized token  http   was expecting   true    false  or  null    at  Source  java io ByteArrayInputStream 20d7aa52  line  1  column  9      at org kitesdk morphline stdio AbstractParser doProcess AbstractParser java 98      at org kitesdk morphline base AbstractCommand process AbstractCommand java 156      at org kitesdk morphline stdlib TryRulesBuilder TryRules doProcess TryRulesBuilder java 120      at org kitesdk morphline base AbstractCommand process AbstractCommand java 156      at org kitesdk morphline base AbstractCommand doProcess AbstractCommand java 181      at org kitesdk morphline base AbstractCommand process AbstractCommand java 156      at org apache flume sink solr morphline MorphlineHandlerImpl process MorphlineHandlerImpl java 128      at org apache flume sink solr morphline MorphlineSink process MorphlineSink java 141      at org apache flume sink DefaultSinkProcessor process DefaultSinkProcessor java 68      at org apache flume SinkRunner PollingRunner run SinkRunner java 147      at java lang Thread run Thread java 744  Caused by  com fasterxml jackson core JsonParseException  Unrecognized token  http   was expecting   true    false  or  null    at  Source  java io ByteArrayInputStream 20d7aa52  line  1  column  9      at com fasterxml jackson core JsonParser  constructError JsonParser java 1524      at com fasterxml jackson core base ParserMinimalBase  reportError ParserMinimalBase java 557      at com fasterxml jackson core json UTF8StreamJsonParser  reportInvalidToken UTF8StreamJsonParser java 3095      at com fasterxml jackson core json UTF8StreamJsonParser  handleUnexpectedValue UTF8StreamJsonParser java 2340      at com fasterxml jackson core json UTF8StreamJsonParser  nextTokenNotInObject UTF8StreamJsonParser java 818      at com fasterxml jackson core json UTF8StreamJsonParser nextToken UTF8StreamJsonParser java 698      at com fasterxml jackson databind MappingIterator hasNextValue MappingIterator java 159      at org kitesdk morphline json ReadJsonBuilder ReadJson doProcess ReadJsonBuilder java 109      at org kitesdk morphline stdio AbstractParser doProcess AbstractParser java 96          10 more

User · Accepted Answer

We have the following string which is a valid JSON ...

Clearly the JSON parser disagrees!

However, the exception says that the error is at "line 1: column 9", and there is no "http" token near the beginning of the JSON. So I suspect that the parser is trying to parse something different than this string when the error occurs.

You need to find what JSON is actually being parsed. Run the application within a debugger, set a breakpoint on the relevant constructor for JsonParseException ... then find out what is in the ByteArrayInputStream that it is attempting to parse.

User · Answer

I faced this exception for a long time and was not able to pinpoint the problem  The exception says line 1 column 9  The mistake I did is to get the first line of the file which flume is processing    Apache flume process the content of the file in patches  So  when flume throws this exception and says line 1  it means the first line in the current patch    If your flume agent is configured to use batch size   100  and  for example  the file contains 400 lines  this means the exception is thrown in one of the following lines 1  101  201 301   How to discover the line which causes the problem   You have three ways to do that   1- pull the source code and run the agent in debug mode  If you are an average developer like me and do not know how to make this  check the other two options   2- Try to split the file based on the batch size and run the flume agent again  If you split the file into 4 files  and the invalid json exists between lines 301 and 400  the flume agent will process the first 3 files and stop at the fourth file  Take the fourth file and again split it into more smaller files  continue the process until you reach a file with only one line and flume fails while processing it    3- Reduce the batch size of the flume agent to only one and compare the number of processed events in the output of the sink you are using  For example  in my case I am using Solr sink  The file contains 400 lines  The flume agent is configured with batch size 100  When I run the flume agent  it fails at some point and throw that exception  At this point check how many documents are ingested in Solr  If the invalid json exists at line 346  the number of documents indexed into Solr will be 345  so the next line is the line which causes the problem    In my case I followed the third option and fortunately I pinpoint the line which causes the problem    This is a long answer but it actually does not solve the exception  How I overcome this exception    I have no idea why Jackson library complain while parsing a json string contains escaped characters  n  r  t  I think  but I am not sure  the Jackson parser is by default escaping these characters which cases the json string to be split into two lines  in case of  n  and then it deals each line as a separate json string    In my case we used a customized interceptor to remove these characters before being processed by the flume agent  This is the way we solved this problem

User · Answer

Add produces    quot application json quot  in  RequestMapping

User · Answer

It might be obvious  but make sure that you are sending to the parser URL object not a String containing www adress  This will not work        ObjectMapper mapper   new ObjectMapper        String www    www sample pl       Weather weather   mapper readValue www  Weather class     But this will       ObjectMapper mapper   new ObjectMapper        URL www   new URL  http   www oracle com         Weather weather   mapper readValue www  Weather class

[java] JsonParseException: Unrecognized token 'http': was expecting ('true', 'false' or 'null')

Examples related to java

Examples related to jackson

Examples related to cloudera